Free the mouse Replay Ndx
Home | Changes | Index | Search | Go

NDX files

Introduction

Each MPEG file, stored as programid.mpg, has a corresponding index file, stored as programid.ndx. The index file includes information purely about the video stream; all information about what program is included, where it came from, and the like, is in the channel guide.

Overall Structure

An index file consists of a 32-byte header, followed by a series of 32-byte records. There's one record per MPEG pack (which, with the Replay's multiplexing, is the same as one record per GOP). That averages out to a bit more than 2 records per second. Some parts of the Replay interface (including the running time shown on the HUD, skipping via the quickstep and instant replay buttons, and random access via NUM quickstep/instant replay/jump buttons) act as if there's exactly one record/pack/GOP every half second.

Vagueness

I have no access to actual source code; this is all based on examination of index files, the corresponding mpeg files, and the like.

Notation

Although I'm using C struct notation, these shouldn't be used as-is; all the items are in big-endian byte order, for example, and need to be converted on little-endian processors. The ReplayPC CVS tree (and the upcoming ReplayPC 0.4 release) includes an ndx-parsing/building library, and some sample applications.

Header

struct ndx_header {
    u8 major_version;    /* 2 */
    u8 minor_version;    /* 2 */
    u8 flags;            /* 0x01 = copy protected; none others seen */
    u8 unused[29];       /* all 0s */
};

Record

struct ndx_record
{
    u8  flag_1;         /* all unknown; different between 520410600 and
                           520411140.  Update: looked like a single flag in
                           520410600, but appears unused & uninitialized in
                           520411140 and the very first software release */
    u8  comm_flag;      /* 0x01 == part of a commercial break; 0x02 ==
                           candidate to be the beginning or ending of a
                           commercial break */
    u16 video_offset;   /* offset, relative to the stream_position member, of
                           the start of the first video PES packet in this
                           pack */
    u8  unk_fe;         /* always 0xfe as far as I've seen; I think this is
                           read directly from one of the mpeg encoder's
                           registers, but I don't know what it means */
    u8  macrovision;    /* always 0 or 3 as far as I've seen; I think this too
                           is read directly from one of the mpeg encoder's
                           registers, and '3' means 'macrovision possibly
                           detected' */
    u16 macrovision_count; /* counts the number of consecutive frames with
                           macrovision fields equal to 3; I think if this
                           reaches 361 (more than 3 consecutive minutes), then
                           the 'copy-protected' flag is set -- details on when the flag is set
                           have changed in 4.1.1 */
    u32 audio_offset;   /* offset, relative to the video PES pointed at by
                           video_offset, of the beginning of the first audio
                           PES in the pack */
    u32 unused;         /* always 0; related somehow to the timestamp, it
                           looks like */
    u64 timestamp;      /* timestamp in nanoseconds, from an unknown base,
                           possibly system startup time.  What *exactly* it's
                           a timestamp of is still somewhat in question; it
                           doesn't match up with any of the timestamps in the
                           MPEG file.  I think it's probably the time the
                           record was written, as measured by a clock that
                           isn't exactly synchronous with the MPEG encoder's
                           clock */
    u64 stream_position; /* the offset into the MPEG file of the start of the
                           32k cluster the pack this record is about starts
                           in. */
};

Bugs

About 5% of the time, the video_offset field (and thus the audio_offset field) is incorrect; in nearly all of those cases, it's a duplicate of the following record's field. The the cases when it's not, that field's video_offset is incorrect as well. These errors are much more likely on records that correspond to very short packs than longer ones. I'm pretty sure this is caused by a race condition; these offsets are being read from locations that have already been updated for the next record, before this record has been written.

-- ToddLarason - 09 Mar 2002


There is also a possibility that the .ndx is correct and the .mpg has errors in it.

-- LeeThompson - 01 Sep 2002


NOTE: Replay5000s have a different version: 3 and 0. ndxdump freaks out a lot so I'd say that the format has changed significantly for Replay5000s.

These seem totally different on the 5000. Right now I'm guessing that the header is much larger judging form the amount of duplicate information between different .NDX files. I'm guessing 40-48 bytes.

-- LeeThompson - 23 Dec 2002


I have a 5040 and have looked at a couple of .NDX files for some 30 minute shows.

The first 32 bytes of the .NDX file is unknown to me as of yet.

However, I have found that everything after that seems to be in 24 byte records. Like so:

struct ndx5000_record {
  u64 timestamp;    /* 8 byte timestamp, in nanoseconds */
  u32 empty1;       /* apparently empty(?)
                       could be part of the video packet offset,
                       to accomodate some insanely large files */
  u32 video_offset; /* video stream packet offset */
  u32 video_length; /* video stream packet length(?) */
  u32 empty2;       /* apparently empty(?) */
};

Those two empty 32-bit words could be used for something else. Like I said earlier, I only analyzed one or two .NDX files.

Hope that helps.

-- TimLee - 13 Jan 2003


You're farther along than I am. About all I'm sure about is BYTE 1 and BYTE 2 of the header is some kind of version number. :)

So you're thinking this has the MPEG packet offsets in it for some reason? Damn I wish I knew what they were using this file for now.

(I'll admit right here, the new NDX file just confuses me. :) )

-- LeeThompson - 15 Jan 2003


Hmm your index_record is 24 bytes not 16 :)

I think the header is 40 bytes or so... have no idea what the entries actually are so it's a mass of unknowns.

-- LeeThompson - 15 Jan 2003


I'd not be surprised if it's all empty except the version, and is sized so that it'll be read as otherwise-all-0s by the ndx readers for all prvious versions.

-- ToddLarason - 15 Jan 2003


Oh it's not empty. Just don't know what the numbers are used for.

-- LeeThompson - 15 Jan 2003


oh. I should really get my obsession turned back on and look at some of these files again...

-- ToddLarason - 15 Jan 2003


Oops... I guess I need to work on my math :-(

-- TimLee - 15 Jan 2003


I hadn't seen any of this work, I'm glad I found it. The other morning at 6am I took a look and what I found. (I don't have notes on me, but what I remember is:

Header size is 14 bytes, after that blocks of size 48 bytes. The nano counter took about 3 seconds to find.

I'll look at the data again now that I see what's been posted here.

-- TWikiGuest - 14 Feb 2003


TimLee? is correct, the header is 16 bytes, each record is 24 bytes, I believe empty1 is actually part of the video_offset (making it 64 bits). video_position would be a better name for it. I guess I'll have to record a > 4Gb show to verify. video_length is actually audio_offset but defined as an offset from video_position. empty2 is likely to hold things like the commercial flag but I cannot tell yet. Anyway, hope this help...

-- TWikiGuest - 29 May 2003


ReplayTV 5000 Series NDX Files

struct ndx5000_record {
  u64 timestamp;            /* 8 byte timestamp, in nanoseconds */
  u64 filepos_iframe;       /* File position to an I-frame */
  u32 iframe_size;         /* Size of the I-Frame (including PES Headers) */
  u32 empty;               /* Always Zero, possibly for alignment */
};

(Thanks to Anon!)

-- LeeThompson - 29 Jun 2003


Is it correct to guess that the 4xxx and 5xxx streaming incompatibility is due (at least in part) to this? Or are the .mpg files also different?

-- TWikiGuest - 28 Jul 2003


Your post will appear before this form in chronological order (newest at bottom)

Topic Ndx . { Edit | Attach | Ref-By | Printable | Diffs | r1.22 | > | r1.21 | > | r1.20 | More }
Revision r1.22 - 30 Jul 2003 - 06:17 GMT - LeeThompson
Parents: WebHome
Copyright © 2001 by the contributing authors. All material on this collaboration tool is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback.