NDX files
Each MPEG file, stored as programid.mpg, has a corresponding
index file, stored as programid.ndx. The index file includes
information purely about the video stream; all information about what program
is included, where it came from, and the like, is in the channel guide.
An index file consists of a 32-byte header, followed by a series of 32-byte
records. There's one record per MPEG pack (which, with the Replay's
multiplexing, is the same as one record per GOP). That averages out to a bit
more than 2 records per second. Some parts of the Replay interface (including
the running time shown on the HUD, skipping via the quickstep and instant
replay buttons, and random access via NUM quickstep/instant
replay/jump buttons) act as if there's exactly one record/pack/GOP every half
second.
I have no access to actual source code; this is all based on examination of
index files, the corresponding mpeg files, and the like.
Although I'm using C struct notation, these shouldn't be used as-is;
all the items are in big-endian byte order, for example, and need to
be converted on little-endian processors. The
ReplayPC CVS tree (and the
upcoming ReplayPC 0.4 release) includes an ndx-parsing/building
library, and some sample applications.
struct ndx_header {
u8 major_version; /* 2 */
u8 minor_version; /* 2 */
u8 flags; /* 0x01 = copy protected; none others seen */
u8 unused[29]; /* all 0s */
};
struct ndx_record
{
u8 flag_1; /* all unknown; different between 520410600 and
520411140. Update: looked like a single flag in
520410600, but appears unused & uninitialized in
520411140 and the very first software release */
u8 comm_flag; /* 0x01 == part of a commercial break; 0x02 ==
candidate to be the beginning or ending of a
commercial break */
u16 video_offset; /* offset, relative to the stream_position member, of
the start of the first video PES packet in this
pack */
u8 unk_fe; /* always 0xfe as far as I've seen; I think this is
read directly from one of the mpeg encoder's
registers, but I don't know what it means */
u8 macrovision; /* always 0 or 3 as far as I've seen; I think this too
is read directly from one of the mpeg encoder's
registers, and '3' means 'macrovision possibly
detected' */
u16 macrovision_count; /* counts the number of consecutive frames with
macrovision fields equal to 3; I think if this
reaches 361 (more than 3 consecutive minutes), then
the 'copy-protected' flag is set -- details on when the flag is set
have changed in 4.1.1 */
u32 audio_offset; /* offset, relative to the video PES pointed at by
video_offset, of the beginning of the first audio
PES in the pack */
u32 unused; /* always 0; related somehow to the timestamp, it
looks like */
u64 timestamp; /* timestamp in nanoseconds, from an unknown base,
possibly system startup time. What *exactly* it's
a timestamp of is still somewhat in question; it
doesn't match up with any of the timestamps in the
MPEG file. I think it's probably the time the
record was written, as measured by a clock that
isn't exactly synchronous with the MPEG encoder's
clock */
u64 stream_position; /* the offset into the MPEG file of the start of the
32k cluster the pack this record is about starts
in. */
};
About 5% of the time, the video_offset field (and thus the audio_offset
field) is incorrect; in nearly all of those cases, it's a duplicate of the
following record's field. The the cases when it's not, that field's
video_offset is incorrect as well. These errors are much more likely on
records that correspond to very short packs than longer ones. I'm pretty sure
this is caused by a race condition; these offsets are being read from
locations that have already been updated for the next record, before this
record has been written.
-- ToddLarason - 09 Mar 2002
There is also a possibility that the .ndx is correct and the .mpg has errors in it.
-- LeeThompson - 01 Sep 2002
NOTE: Replay5000s have a different version: 3 and 0. ndxdump freaks out a lot so I'd say that the format has changed significantly for Replay5000s.
These seem totally different on the 5000. Right now I'm guessing that the header is much larger judging form the amount of duplicate information between different .NDX files. I'm guessing 40-48 bytes.
-- LeeThompson - 23 Dec 2002
I have a 5040 and have looked at a couple of .NDX files for some 30 minute shows.
The first 32 bytes of the .NDX file is unknown to me as of yet.
However, I have found that everything after that seems to be in 24 byte records. Like so:
struct ndx5000_record {
u64 timestamp; /* 8 byte timestamp, in nanoseconds */
u32 empty1; /* apparently empty(?)
could be part of the video packet offset,
to accomodate some insanely large files */
u32 video_offset; /* video stream packet offset */
u32 video_length; /* video stream packet length(?) */
u32 empty2; /* apparently empty(?) */
};
Those two empty 32-bit words could be used for something else. Like I said earlier, I only analyzed one or two .NDX files.
Hope that helps.
-- TimLee - 13 Jan 2003
You're farther along than I am. About all I'm sure about is BYTE 1 and BYTE 2 of the header is some kind of version number. :)
So you're thinking this has the MPEG packet offsets in it for some reason? Damn I wish I knew what they were using this file for now.
(I'll admit right here, the new NDX file just confuses me. :) )
-- LeeThompson - 15 Jan 2003
Hmm your index_record is 24 bytes not 16 :)
I think the header is 40 bytes or so... have no idea what the entries actually are so it's a mass of unknowns.
-- LeeThompson - 15 Jan 2003
I'd not be surprised if it's all empty except the version, and is sized so that it'll be read as otherwise-all-0s by the ndx readers for all prvious versions.
-- ToddLarason - 15 Jan 2003
Oh it's not empty. Just don't know what the numbers are used for.
-- LeeThompson - 15 Jan 2003
oh. I should really get my obsession turned back on and look at some of these files again...
-- ToddLarason - 15 Jan 2003
Oops... I guess I need to work on my math :-(
-- TimLee - 15 Jan 2003
I hadn't seen any of this work, I'm glad I found it. The other morning at 6am I took a look and what I found. (I don't have notes on me, but what I remember is:
Header size is 14 bytes, after that blocks of size 48 bytes.
The nano counter took about 3 seconds to find.
I'll look at the data again now that I see what's been posted here.
-- TWikiGuest - 14 Feb 2003
TimLee? is correct, the header is 16 bytes, each record is 24 bytes,
I believe empty1 is actually part of the video_offset (making it 64 bits). video_position would be a better name for it. I guess I'll have to record a > 4Gb show to verify.
video_length is actually audio_offset but defined as an offset from video_position.
empty2 is likely to hold things like the commercial flag but I cannot tell yet. Anyway, hope this help...
-- TWikiGuest - 29 May 2003
ReplayTV 5000 Series NDX Files
struct ndx5000_record {
u64 timestamp; /* 8 byte timestamp, in nanoseconds */
u64 filepos_iframe; /* File position to an I-frame */
u32 iframe_size; /* Size of the I-Frame (including PES Headers) */
u32 empty; /* Always Zero, possibly for alignment */
};
(Thanks to Anon!)
-- LeeThompson - 29 Jun 2003
Is it correct to guess that the 4xxx and 5xxx streaming incompatibility is due (at least in part) to this? Or are the .mpg files also different?
-- TWikiGuest - 28 Jul 2003
Revision r1.22 - 30 Jul 2003 - 06:17 GMT - LeeThompson Parents: WebHome
|
Copyright © 2001 by the contributing authors.
All material on this collaboration tool is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback.
|
|