Does anybody what the file system on the Digifusion hard disk is? Will Windows XP professional see it without creating a partition?
I'm not sure who created it, but it doesn't conform to a PC's view of how a HD should look. There is no partition table so the PC thinks it has rubbish on it and windows will offer to re-initialise it (DO NOT LET IT!)
So, sorry, you need either Syphon or Daniels extractor to get data off. No way to put stuff back on as yet (unless you fancy manually editting the sectors!)
K
Yup I have a C:\ too - not much room on it - but should be no problem creating a C:\fvrt100 there.
Cliff
This is the problem!!!!
Ye Ha!
Create the folder fvrt100 off your C: drive and all should be well.
I'll make a drop tonight to sort this out.
I haven't a clue why people using IDE rather than USB i/f are affected more, but I have just re-created it using a usb caddy.
It uses the folder to dump the channel data and the debug file.
I'm Daniel Hope - the guy who wrote "the other tool" for extracting programs of your Fusion harddrive. I've finally got around to posting...
As Karl mentioned earlier in this thread, I have been playing around with the firmware files and have managed to decompress them. There's a bunch of interesting stuff inside (well... how interesting depends on how interested you are in embedded systems!).
I've written a wee tool for others to decompress their firmware images if they are curious too - see http://www.hope.co.nz/projects/fusion/ for details - look for the "Firmware Utility". You will need to point it at an extracted run.bin file that you have previously taken off your harddrive.
Inside there is a whole bunch of code that I guess runs directly on the STi5514 - it's probably possible to disassemble it using one of the ST20 disassemblers out on the internet.
Interestingly there were also some PNG files embedded - I was hopeful that they would be the background files for the EPG - with the hope being that I could hack it to a plainer background. Unfortunately the PNG files embedded seem to be something to do with an MHEG application :-( The background files may be stored in ROM (in which case they cant be changed), or possibly stored in a bitmap format (like the .tnj thumbnail preview files) which are a bit harder to find in the file.
Anyway, if you are interested, have a play with it and let me know how you get on.
On another note - I haven't really had much time to work on my extractor program recently (been pretty busy at work), but I am hoping to get out an update in the near future that will support Thomson drives better. Apologies for those of you that have emailed me, but not received a reply regarding this.
Great to have both experts in one place. While I was having problems with Karl's program I was using yours but there were some of my PES files where it would just show the green progress bar to about 50% say having copied 1.25GB of a 2.5GB file and then it just stops with no reason given.
I also had some files that every time I tried to extract it caused a 0xC00005 exception and the program just exited with the "send bug report to M$?" prompt.
The same files were extracted OK with Karl's program once we got past the start-up problems.
slightly off topic. I emailed fusion last week and asked them if there was a possibility of removing the hideous background and replacing it with a plainer one like $ky +. I received a reply that they would pass on my coments but that there were no immediate plans change the background.
I seem to remember that a while back fusion were supposed to be changing the image to a higher resolution one (it's mentioned somewhere in the main fusion thread I think) so it may be possible.
Perhaps if more people asked for this feature.....
please keep up the excellent work on these programs for extracting data!
I've written a wee tool for others to decompress their firmware images if they are curious too
Had a play, contents are very interesting, I can see the png files there also appears to be an encoded video stream in there too. Will try and extract that later.....
EDIT** @Dan, The backdrops are stored in the m2v file as a series of frames :-)
Also great to see Cliff in there with the coding tweaks . I left programming alone years ago when I discovered that depressingly, I could achieve the same thing in three lines of BBC BASIC as I had written in 80 lines of 6502 assembly language :rolleyes:. And when the Acorn RISC Machine came out I sooooo wanted to learn the assembler with that plethora of registers, but sadly, couldn't afford the RISC PC...
Anyway, it's great to see more heads getting together on this one. I have been a bit quiet on the posting front, but nevertheless am still engaged in trying to sort out these rogue Thomson recordings. Here's what I found so far:
a) The same recordings that are problematic with Dan's extractor (ie when the program just stops transferring data) also cause problems when cleaning up with Project X, even though the recording may be successfully extracted with Syphon. As I alluded to before, it seems to be a point in the recording where the timing data of audio and video packets becomes corrupted, resulting in serious problems with synchronising the two. In these files, Project X will only recover the same amount of material from a Syphon extraction as is extracted by Dan's program. It seems unlikely that two independently written programs would make the same exact errors, so the conclusion is probably that it's something about the Thomson recordings that is very strange.
b) I made two new 5 min recordings with the Thomson; the first just a straight uncomplicated recording made by pressing the 'rec' button to dump the buffer + a few extra mins recording, then the second which was triggered in the same way but this time with plenty of use of the pause/rewind/ffwd/live buttons to see whether that made a difference. Extraction of both recordings was attempted with Syphon and Dan's extractor, and successful extractions cleaned up with PX. As before, the problematic file was the same with both extraction apps, and surprisingly was the 'straight' recording. In fact the one in which the the pause/rewind/ffwd/live buttons were used during recording was processed perfectly by PX, and extracted by Dan's prog. Looking at the log output from PX showed there were no audio/video errors.
c) I've replaced the drive in the Thomson and used the navigation controls to see whether that alters anything in both recordings, but sadly it doesn't.
d) Now I'm wondering whether it's the first recording that the Thomson makes, after an EPG update, that generates problems. I'll carry on to try and see a pattern. Meanwhile there are some settings in PX which will get the AV data successfully from a Syphon extraction of a problematic recording, but it's tediously slow at a plateau of 12 fps and the sync is horrible afterwards. It's easier to just to an analogue copy to DVD in real-time via the RGB SCART.
Ho hum - will report back in another couple of years!
Yes, I can recompress the file, but unfortunately, I don't know how to recalculate what i think is a checksum in the header of the firmware file. I am guessing from the boot sequence that you can see on the serial port, that the checksum is calculated on the compressed data only (or at least a portion of it)
I haven't tried, but I would guess that an invalid checksum will result in a the bootloader reverting to the firmware in ROM.
The checksum algorithm would be embedded into the bootloader that we dont have ready access to - so it would not be possible to reverse engineer from that point of view. However, the uncompressed firmware file contains code that will download OTA updates - it may check it there, but that would involve stepping through a bunch of ST20 assembler (which definately doesn't have the easiest assembler to understand...)
I'm scratching my head a bit on how the checksum could be calculated - I'll upload a tool shortly that will allow you to recompress the file and manually specify the fields. If you are interested the compressed data is in zlib format (I took a wild guess at this, and was surprised when zlib managed to extract it!), so you can fairly simply build a tool to do the decompression/compression yourself using the code at http://www.zlib.net/
I'm scratching my head a bit on how the checksum could be calculated - I'll upload a tool shortly that will allow you to recompress the file and manually specify the fields. If you are interested the compressed data is in zlib format (I took a wild guess at this, and was surprised when zlib managed to extract it!), so you can fairly simply build a tool to do the decompression/compression yourself using the code at http://www.zlib.net/
Thanks Dan,
I'll take a look over the next few nights and see if I can come up with something....
:-)
K
There's got to be a strong chance that they are using a "standard" CRC algorithm. It may be worth Googling for any/all of LRC, CRC16, CRC32, CCITT, SumCheck, XOR
but then again they might even be putting it through an MD5 or an SHA perhaps?
I keep meaning to make the following feature request for Syphon:
As you probably know, ITV have a nasty habit of showing films split across the 10 o'clock news. So in the file list in Syphon you may find you have two items called:
(17_11_05) Living Daylights
(17_11_05) Living Daylights
Before I realised this I copied about 3GB for the first half of Titanic from the box, then copied the other half but only later realised that it had just over-written the first extraction.
Do you think it could maybe try to open the target file first and if present either show a "are you sure you want to over-write?" dialog or perhaps just suffix the new filename _#2 or something?
Also, while the date display in the PES list is nice it would be better (IMHO anyway) if, when the file was extracted if the "(dd_mm_yy) " prefix could be dropped from the resultant file.
So my two entries above would lead to:
Living Daylights.mp2
Living Daylights.m2v
Living Daylights_#2.mp2
Living Daylights_#2.m2v
Do you think it could maybe try to open the target file first and if present either show a "are you sure you want to over-write?" dialog or perhaps just suffix the new filename _#2 or something?
The original reason for adding the date prefix was to avoid this problem when storing weeklies (Dr Who in my case).
I was looking at allowing a format specifier to be used to allow people to choose their own layout (but got a little distracted over the access violation bug) I'll take another look at it. I didn't think of the possibility of two recordings having the same name on the same date....
I think it is possible to edit the name in the tree view. Just double click on the name and you should be able to change it to whatever you want.... I cannot test this here (I'm going to have to clone the test 8Gb drive I have used in the past onto my laptop!)...
Also, while the date display in the PES list is nice it would be better (IMHO anyway) if, when the file was extracted if the "(dd_mm_yy) " prefix could be dropped from the resultant file.
Cliff
Fine for films, not for people wanting to archive series... I think the format specifier and a check for existance will cover most bases... Something to keep me busy on these cold winter nights!!
I have tried the threading you mentioned in a previous post and it works (the main window is re-drawn) but I will have to work out what to do about the progress bars....
Any improvements, recommendations, are welcome. It all goes to giving me C++ experience and others something useful!
K
There's got to be a strong chance that they are using a "standard" CRC algorithm. It may be worth Googling for any/all of LRC, CRC16, CRC32, CCITT, SumCheck, XOR
but then again they might even be putting it through an MD5 or an SHA perhaps?
Cliff
Cliff has hit the nail on the head, the CRC is CRC32 on the compressed file without the final (~ bit inverting).
You know, I think we will end up with some pretty funky tools very soon!
K
At this stage the crap background is about the ONLY thing wrong with the Fusion. If you chaps come up with a simple utility to re-write run.bin without this then it'll make it one of the best PVRs out there!
(might even persuade my parents to use it a bit more often if they could actually READ the menus!)
At this stage the crap background is about the ONLY thing wrong with the Fusion. If you chaps come up with a simple utility to re-write run.bin without this then it'll make it one of the best PVRs out there!
(might even persuade my parents to use it a bit more often if they could actually READ the menus!)
Cliff
The bits are all there, just need the glue.
The problem will be to convert a backdrop to m2v format that the PVR will accept...
I have tried the threading you mentioned in a previous post and it works (the main window is re-drawn) but I will have to work out what to do about the progress bars....
In my code for processing HTTP logs I simply have the following in the main processing thread:
//
// Update status bar
//
if ( line_guess > 100 )
if ( lines_read % (line_guess / 100) == 0 )
{
//
//Every 100th of the file inc. progress bar
//
percent_done += 1;
if ( percent_done < 101 )
SendMessage( hWndPB, PBM_STEPIT, 0, 0 );
}
The key thing being that the hWnd to the Progress Bar is a global that is visible to this thread. So it just sends (not posts) it messages to get the progress moved on.
Cliff has hit the nail on the head, the CRC is CRC32 on the compressed file without the final (~ bit inverting).
Hi Karl,
I can't seem to get this to work - I guess there must be something screwy with my CRC32 implementation... Even if I don't do the final bit inversion, it still doesn't add up.
For example, version 1.7.4 (haven't got round to yanking 1.7.5 off the box yet)
Compressed Size: 0x000F5024
CRC32: 0x18D89879
CRC32 (no inversion): 0xE7276786
Checksum in header: 0x2F7E9E52
Any clues? What data range are you using for the checksum? - do you include the header too?
Hi Karl,
Any clues? What data range are you using for the checksum? - do you include the header too?
Dan
Hi Dan,
The file header appears to me to look like
{
unsigned int compressedSize;
unsigned int uncompressedSize;
unsigned int majorRevNum;
unsigned int minorRevNum;
unsigned int checkSum;
unsigned int blank[3];
}
should be 32bytes. I tested mine on a 1.7.5 someone sent me....
Take a trip to the Thomson extraction forum (same tools different PVR) ... Look up posts from Musukebba...
Actually I posted that before the separate "Extraction" threads got split off. The original was here - if it seems to be useful I'll repost it over at the Thomson extraction thread.
Comments
I'm not sure who created it, but it doesn't conform to a PC's view of how a HD should look. There is no partition table so the PC thinks it has rubbish on it and windows will offer to re-initialise it (DO NOT LET IT!)
So, sorry, you need either Syphon or Daniels extractor to get data off. No way to put stuff back on as yet (unless you fancy manually editting the sectors!)
K
This is the problem!!!!
Ye Ha!
Create the folder fvrt100 off your C: drive and all should be well.
I'll make a drop tonight to sort this out.
I haven't a clue why people using IDE rather than USB i/f are affected more, but I have just re-created it using a usb caddy.
It uses the folder to dump the channel data and the debug file.
Now to fix it!
I just created a c:\fvrt100 ran the program and it came up without any problems and I found two files in \fvrt100 (chantab.dat and filedata.txt)
Cliff
Fixed access violation bug
Moves debug from a file to a tab
Stop unwanted generation of chantab.dat file
I'm Daniel Hope - the guy who wrote "the other tool" for extracting programs of your Fusion harddrive. I've finally got around to posting...
As Karl mentioned earlier in this thread, I have been playing around with the firmware files and have managed to decompress them. There's a bunch of interesting stuff inside (well... how interesting depends on how interested you are in embedded systems!).
I've written a wee tool for others to decompress their firmware images if they are curious too - see http://www.hope.co.nz/projects/fusion/ for details - look for the "Firmware Utility". You will need to point it at an extracted run.bin file that you have previously taken off your harddrive.
Inside there is a whole bunch of code that I guess runs directly on the STi5514 - it's probably possible to disassemble it using one of the ST20 disassemblers out on the internet.
Interestingly there were also some PNG files embedded - I was hopeful that they would be the background files for the EPG - with the hope being that I could hack it to a plainer background. Unfortunately the PNG files embedded seem to be something to do with an MHEG application :-( The background files may be stored in ROM (in which case they cant be changed), or possibly stored in a bitmap format (like the .tnj thumbnail preview files) which are a bit harder to find in the file.
Anyway, if you are interested, have a play with it and let me know how you get on.
On another note - I haven't really had much time to work on my extractor program recently (been pretty busy at work), but I am hoping to get out an update in the near future that will support Thomson drives better. Apologies for those of you that have emailed me, but not received a reply regarding this.
Cheers,
Dan Hope
Congratulations on the decompression tool, I'll have a play with it tonight!!
Cheers
Karl
Great to have both experts in one place. While I was having problems with Karl's program I was using yours but there were some of my PES files where it would just show the green progress bar to about 50% say having copied 1.25GB of a 2.5GB file and then it just stops with no reason given.
I also had some files that every time I tried to extract it caused a 0xC00005 exception and the program just exited with the "send bug report to M$?" prompt.
The same files were extracted OK with Karl's program once we got past the start-up problems.
Cliff
slightly off topic. I emailed fusion last week and asked them if there was a possibility of removing the hideous background and replacing it with a plainer one like $ky +. I received a reply that they would pass on my coments but that there were no immediate plans change the background.
I seem to remember that a while back fusion were supposed to be changing the image to a higher resolution one (it's mentioned somewhere in the main fusion thread I think) so it may be possible.
Perhaps if more people asked for this feature.....
please keep up the excellent work on these programs for extracting data!
Dweeb.
Had a play, contents are very interesting, I can see the png files there also appears to be an encoded video stream in there too. Will try and extract that later.....
EDIT**
@Dan, The backdrops are stored in the m2v file as a series of frames :-)
Do you have a way to re-compress the file ????
K
Also great to see Cliff in there with the coding tweaks
Anyway, it's great to see more heads getting together on this one. I have been a bit quiet on the posting front, but nevertheless am still engaged in trying to sort out these rogue Thomson recordings. Here's what I found so far:
a) The same recordings that are problematic with Dan's extractor (ie when the program just stops transferring data) also cause problems when cleaning up with Project X, even though the recording may be successfully extracted with Syphon. As I alluded to before, it seems to be a point in the recording where the timing data of audio and video packets becomes corrupted, resulting in serious problems with synchronising the two. In these files, Project X will only recover the same amount of material from a Syphon extraction as is extracted by Dan's program. It seems unlikely that two independently written programs would make the same exact errors, so the conclusion is probably that it's something about the Thomson recordings that is very strange.
b) I made two new 5 min recordings with the Thomson; the first just a straight uncomplicated recording made by pressing the 'rec' button to dump the buffer + a few extra mins recording, then the second which was triggered in the same way but this time with plenty of use of the pause/rewind/ffwd/live buttons to see whether that made a difference. Extraction of both recordings was attempted with Syphon and Dan's extractor, and successful extractions cleaned up with PX. As before, the problematic file was the same with both extraction apps, and surprisingly was the 'straight' recording. In fact the one in which the the pause/rewind/ffwd/live buttons were used during recording was processed perfectly by PX, and extracted by Dan's prog. Looking at the log output from PX showed there were no audio/video errors.
c) I've replaced the drive in the Thomson and used the navigation controls to see whether that alters anything in both recordings, but sadly it doesn't.
d) Now I'm wondering whether it's the first recording that the Thomson makes, after an EPG update, that generates problems. I'll carry on to try and see a pattern. Meanwhile there are some settings in PX which will get the AV data successfully from a Syphon extraction of a problematic recording, but it's tediously slow at a plateau of 12 fps and the sync is horrible afterwards. It's easier to just to an analogue copy to DVD in real-time via the RGB SCART.
Ho hum - will report back in another couple of years!
Yes, I can recompress the file, but unfortunately, I don't know how to recalculate what i think is a checksum in the header of the firmware file. I am guessing from the boot sequence that you can see on the serial port, that the checksum is calculated on the compressed data only (or at least a portion of it)
I haven't tried, but I would guess that an invalid checksum will result in a the bootloader reverting to the firmware in ROM.
The checksum algorithm would be embedded into the bootloader that we dont have ready access to - so it would not be possible to reverse engineer from that point of view. However, the uncompressed firmware file contains code that will download OTA updates - it may check it there, but that would involve stepping through a bunch of ST20 assembler (which definately doesn't have the easiest assembler to understand...)
I'm scratching my head a bit on how the checksum could be calculated - I'll upload a tool shortly that will allow you to recompress the file and manually specify the fields. If you are interested the compressed data is in zlib format (I took a wild guess at this, and was surprised when zlib managed to extract it!), so you can fairly simply build a tool to do the decompression/compression yourself using the code at http://www.zlib.net/
Cheers,
Dan
Thanks Dan,
I'll take a look over the next few nights and see if I can come up with something....
:-)
K
There's got to be a strong chance that they are using a "standard" CRC algorithm. It may be worth Googling for any/all of LRC, CRC16, CRC32, CCITT, SumCheck, XOR
but then again they might even be putting it through an MD5 or an SHA perhaps?
Cliff
I keep meaning to make the following feature request for Syphon:
As you probably know, ITV have a nasty habit of showing films split across the 10 o'clock news. So in the file list in Syphon you may find you have two items called:
(17_11_05) Living Daylights
(17_11_05) Living Daylights
Before I realised this I copied about 3GB for the first half of Titanic from the box, then copied the other half but only later realised that it had just over-written the first extraction.
Do you think it could maybe try to open the target file first and if present either show a "are you sure you want to over-write?" dialog or perhaps just suffix the new filename _#2 or something?
Also, while the date display in the PES list is nice it would be better (IMHO anyway) if, when the file was extracted if the "(dd_mm_yy) " prefix could be dropped from the resultant file.
So my two entries above would lead to:
Living Daylights.mp2
Living Daylights.m2v
Living Daylights_#2.mp2
Living Daylights_#2.m2v
Cliff
The original reason for adding the date prefix was to avoid this problem when storing weeklies (Dr Who in my case).
I was looking at allowing a format specifier to be used to allow people to choose their own layout (but got a little distracted over the access violation bug) I'll take another look at it. I didn't think of the possibility of two recordings having the same name on the same date....
I think it is possible to edit the name in the tree view. Just double click on the name and you should be able to change it to whatever you want.... I cannot test this here (I'm going to have to clone the test 8Gb drive I have used in the past onto my laptop!)...
Fine for films, not for people wanting to archive series... I think the format specifier and a check for existance will cover most bases... Something to keep me busy on these cold winter nights!!
I have tried the threading you mentioned in a previous post and it works (the main window is re-drawn) but I will have to work out what to do about the progress bars....
Any improvements, recommendations, are welcome. It all goes to giving me C++ experience and others something useful!
K
Cliff has hit the nail on the head, the CRC is CRC32 on the compressed file without the final (~ bit inverting).
You know, I think we will end up with some pretty funky tools very soon!
K
At this stage the crap background is about the ONLY thing wrong with the Fusion. If you chaps come up with a simple utility to re-write run.bin without this then it'll make it one of the best PVRs out there!
(might even persuade my parents to use it a bit more often if they could actually READ the menus!)
Cliff
The bits are all there, just need the glue.
The problem will be to convert a backdrop to m2v format that the PVR will accept...
// // Update status bar // if ( line_guess > 100 ) if ( lines_read % (line_guess / 100) == 0 ) { // //Every 100th of the file inc. progress bar // percent_done += 1; if ( percent_done < 101 ) SendMessage( hWndPB, PBM_STEPIT, 0, 0 ); }The key thing being that the hWnd to the Progress Bar is a global that is visible to this thread. So it just sends (not posts) it messages to get the progress moved on.BTW, when I stop processing I just: and in the init of the program I:
Cliff
Hi Karl,
I can't seem to get this to work - I guess there must be something screwy with my CRC32 implementation... Even if I don't do the final bit inversion, it still doesn't add up.
For example, version 1.7.4 (haven't got round to yanking 1.7.5 off the box yet)
Compressed Size: 0x000F5024
CRC32: 0x18D89879
CRC32 (no inversion): 0xE7276786
Checksum in header: 0x2F7E9E52
Any clues? What data range are you using for the checksum? - do you include the header too?
Dan
Hi Dan,
The file header appears to me to look like
{
unsigned int compressedSize;
unsigned int uncompressedSize;
unsigned int majorRevNum;
unsigned int minorRevNum;
unsigned int checkSum;
unsigned int blank[3];
}
should be 32bytes. I tested mine on a 1.7.5 someone sent me....
I'll mail you my CRC32 routine later...
K
Can anybody please tell me what I do with ProjectX in order to clean these files up before multiplexing them?
You are welcome
Take a trip to the Thomson extraction forum (same tools different PVR):
Look up posts from Musukebba, he has done a shedload of research into this.
http://forum.digitalspy.co.uk/board/showthread.php?t=297714
PM'ed it to you, managed to misplace your e-mail address!
K