DetectAds - Eliminate watching the commercial breaks!
DetectAds analyses a recording to detect the commercial breaks and places bookmarks at the start (optional) and end of each break so that you can just use the Bookmark button (arrowed in picture) to jump instantly to the next bookmark. Alternatively you can leave the remote languishing between the sofa cushions and have DetectAds physically remove the commercials and EPG data from the recording shrinking the recording size considerably.
- 1 DetectAds - Eliminate watching the commercial breaks!
- 1.1 Overview
- 1.2 Installation
- 1.3 Automatic Processing
- 1.4 Invoking from Browse or FlexView
- 1.5 Command Line
- 1.6 Settings
- 1.7 Chase playing
- 1.8 Limitations
- 1.9 Debugging
- 1.10 Under the covers
- 1.11 Change History
- 1.12 Future Enhancements
- 1.13 Acknowledgements
DetectAds is in the Packages catalogue. Chaseget, ir and some other packages used by DetectAds will be installed/upgraded automatically as needed.
Traditionally automatic webif packages have been run sequentially on completed recordings under the control of the "auto" process which ran at fixed intervals. DetectAds can be run in this mode triggered from auto when auto-decryption completes.
Recent enhancements have improved the speed of this post-processing by triggering auto when a recording ends and allowing decryption to start before the system has DLNA indexed the recording but it still takes a significant time to sequentially run a number of processes on a recording.
DetectAds is an exceptionally slow process taking about 15 minutes to process an hour of SD recording which means that with the time to decrypt recording and perform crop there is a considerable delay before a fully processed recording is available for viewing.
DetectAds now introduces as an option a radically different processing model, with "Detect whilst recording" (also known as Chaserun) the detection process runs in parallel with the actual recording so the processing completes within a few seconds of the recording finishing. You can even,with some limitations, start to watch your program adfree whilst it is still recording.
With both processing modes you can choose to exclude certain commercial free channels such as BBC and Radio stations from processing (Ads on radio stations are not recognized by DetectAds)
Some users have reported that when the system is exceptionally busy such as when recording multiple HD channels and playing other recordings that running DetectAds can cause Picture and Sound breakups on the current recordings, if you experience such problems you can limit the amount of CPU used by the processor intensive part of DetectAds or / and defer processing until the system is quieter.
Detect whilst recording -Chaserun processing
Ad detection requires a decrypted recording to process so when detecting while recording is in progress there can be three copies of the recording in existence simultaneously: The encrypted input file, the decrypted intermediate file and the cropped & shrunk output file.
To disambiguate between the various files in both the webif and the standard Humax TV interface DetectAds adds the qualifier -dec to the file name and the word Decrypt to the programme title for the decrypted file and adds the qualifier -crop to the file name and the word Crop to the program title for the cropped file.
At the end of recording the input files will normally be deleted and the output renamed to the original file name and title. File renaming will not take place if any of the following is true:
- The "Dustbin/delete original recording after processing?" option is set to NO
- The Input or Output files are Inuse (e.g. being played or processed by another task)
- There is a length error detected (see below)
There is a small chance that there will be a failure in the retrieval of the decrypted file resulting in a file that is longer or shorter than it should be. This seems to occur if the system is DLNA indexing the recording whilst it is being retrieved. Since retrieval is usually within 30 seconds of actual recording it is a rare event and usually only affects the closing credits/ trailing ads. If it does occur the input recording will be left in situ and not replaced by the -dec (which will be deleted) or -crop files (which will be kept, If the length error is significant, more than 30 seconds long, the input file will be automatically queued to retry ad detection since it is unlikely to recur again.
It is also possible that a poor quality recording due to low strength or transmission glitches will cause the detection process to drop frames resulting in an apparently shorter recording leading to misplaced bookmarks and cropping.
If either form of length error is detected a system notification message is generated to warn you that bookmarks and cropping maybe unreliable, the input file is not deleted and -Len err is appended to the title of the -crop file (if auto cropping is active)
The DLNA server is not active in half-awake recording state so ChaseGet uses the IR package to force the system fully awake if needed. After the recording ChaseGet will now (chaseget version 0.1.1_1) attempt to turn power off when it it safe to do so but it is not easy to determine if the user has started to watch the TV in the mean time without pressing any buttons on the remote.
To reduce the risk of upsetting important family members when the box shuts down without warning (The Customised Firmware has no ability to put a 2 minute standby warning on screen like the Humax does) you can specify a time range in the ChaseGet Settings during which automatic return to standby won't operate.
You can also choose in the settings whether to Mute the sound when ChaseGet does wake the system from standby outside the peak time-range. This would prevent a sudden blast of sound if the system wakes up in the middle of the night for a scheduled recording and the TV has been left on or auto-switches on HDMI activity. A user un-muting the sound also indicates that someone is watching the TV and so ChaseGet would not then return the system to standby.
ChaseGet will return the system to StandBy if during scheduled checks which run every 20 minute all of the following are true.
- The system was awakened from Standby by ChaseGet
- The time is not within the exclusion time range
- The system isn't currently in standby
- No significant programs are running (including auto, detectads etc)- full list is in /mod/etc/chaseget.conf and can be extended if needed
- There are no active PuTTY / Telnet psuedo-terminals
- The system hasn't been been rebooted since wakeup
- No recording is being played (on set or via remote access)
- There is no active Reminder event
- No remote control buttons have been pressed since wakeup
Unlike the chaserun processing there is a prerequisite for the file to decrypted first and bookmarks are added to it so there is no output file unless the Crop option is selected, for consistency with chaserun processing deletion of the input depends on the 'Delete input' option. If the input file is not deleted then the output file name is suffixed with -crop.
It is possible that a poor quality recording due to low strength or transmission glitches will cause the detection process to drop frames resulting in an apparently shorter recording leading to misplaced bookmarks. If a significant length error (>5 seconds) is detected a system notification message is generated to warn you that bookmarks maybe unreliable, the recording will not be cropped automatically.
Setting the folder auto-decrypt option is required for the DetectAds process to be trigged but if you are using Flatten or Sweeper to rename/move files between folders it is recommended that the auto-decrypt option only be specified on the target folder to prevent the chance of DetectAds not finding the file because it has been moved.
If you wish wish to defer DetectAds processing to a time when the system is less busy you can specify a target start time on the Setiings page, the target time is the earliest start time but processing may start later depending on when the regular DetectAds task starts and how many other items need to be processed on the queue. You must of course ensure the the system is switched on at the target processing time and remains on long enough to process all of the queued items (possibly set up a scheduled repeating reminder event)
There are two new folder flags Auto DetectAds and No DetectAds
These work in conjunction with the existing Channel exclusions rules so it is not necessary to add the flag to each directory if you use the existing DetctAds Auto processing options (Automatically process whilst recording in progress? (chaserun) or Automatically process recording following auto-decryption? (traditional)
You can also use the Auto DetectAds flag to process exist recordings in a folder.
The Auto DetectAds flag means all recordings in a folder will be processed regardless of the recording channel and whether they have been decrypted. The No DetectAds flag means none of the recordings in a folder will be processed regardless of the recording channel. In the absence of a flag recordings will be processed during recording (chaserun) or following decryption (traditional) if they are on a channel that has not been excluded by the channel filters.
If you routinely want to detect ads in all commercial programmes as I do you don't need to use the new flags at all. If you only want a subset of programmes processed you can use the flags to include/exclude directories overriding the channel filters
For instance you could set No DetectAds on the 'My Videos' root folder to prevent auto ad detection of one-off recordings (I don't recommend setting Auto DetectAds on 'My Videos' since this would override the channel selection filters and cause detection of all one-off recordings including those on BBC channels)
The processing is (deliberately) different from the other auto options where you have to have a flag set before any processing happens.
Another method of automating DetectAds processing is to set up Sweeper rules to select which recording should be processed by DetectAds. This allows you much finer control of the process than the other automation methods that only allow for exclusion based on channel name or folder. Since Sweeper only processes files once they have completed recording and are not in-use it is inevitably slower than using Detect whilst recording.
When the DetectAds package is installed several new options are added to the Sweeper rules menus. These allow you to:
- Check whether the recording has been flagged as 'Ad-Detection done'
- Check whether the recording has already been queued for detection
- Add the recording to the auto processing queue, optionally specifying the time for processing to start,
There is one rule in the set.
The rule may need to be modified to meet your own requirements, as supplied it is a file rule (recursive into child folders) that adds to the auto processing queue recordings that, have no bookmarks, have no Addetection flag set, are not on a BBC channel, and are not already on the queue.
If you have other rules that rename or move the recording file they should precede this rule.
If you wish wish to defer DetectAds processing to a time when the system is less busy you can specify current time conditions in the Sweeper rule to ensure recordings are only queued when you want them processed. You must of course ensure the the system is switched on at the target processing time and remains on long enough to process all of the queued items (possibly set up a scheduled repeating reminder event)
Note: Previous versions of DetectAds provided specific DetectAds queuing actions and conditions which have now been superseded by the standard 'Queue recording for' action, the older actions remain for compatibly with pre-existing rule sets. The -t hh;mm target start time is only supported with the older action.
To crop or not to crop - that is the question
Detecting ads can never be 100% accurate with the risk of it failing to spot an ad break, incorrectly identifying part of a program as an ad break or mis-positioning the end of break markers.
In over a year of using DetectAds I have seen very few detection errors with just a few failures to spot an ad break so I am happy to allow cropping to take place automatically - the original can always be rescued from the dustbin if needed (I do recommend use of Undelete package for safety)
However everybody's viewing habits differs and the success of DetectAds may vary by channel or program time so it is probably worth not using the 'Delete input' option until you are comfortable that cropping is working for you.
If you do not use auto crop the Bookmarks set by DetectAds will now be automatically 'inverted' to meet nicesplice's requirements removing the need to perform that operation on the Crop menu page before performing the Crop operation manually.
Note: Do not use the "Only Bookmark ends of ad breaks?" if you want to retain the ability to run a manual crop,
Invoking from Browse or FlexView
The Addetection icon is now displayed with the other recording flags in the file list.
To process multiple recordings you can select the recordings, choose 'Ad Detection' from the list and then click on the 'Queue For' button at the bottom of the page. Remember Ad Detection is slow so be be sparing when adding batches of existing recordings to the queue.
To process a single recording you can select "Detect Adverts" from the Opt+ pull down menu on the webif Browse file list. This will lead you to the main panel where there are two processing options:
- Run Analysis now
- This processes the file immediately with the results displayed in the processing routine but due to the length of time taken to analyse a recording it is recommended that this option not be used unless you have a lot of spare time or need to investigate a problem!
- Run Analysis in background
- This option adds the recording to the Auto processing queue and returns. The queue is processed sequentially and, if the system is restarted, resumes on restart so if you have a number of recordings to be processed they can be safely added to the queue without waiting for preceding files to finish processing.
If you wish wish to defer DetectAds processing to a time when the system is less busy you can specify a target start time or ASAP on the panel. The target time is the earliest start time but processing may start later depending on when the regular auto processing task starts and how many other items need to be processed on the queue. You must of course ensure the the system is switched on at the target processing time and remains on long enough to process all of the queued items (possibly set up a scheduled repeating reminder event)
If the selected recording is already on the queue the two Run buttons are replaced by:
- Remove from background queue
- This removes the entry from the processing queue but it does not cancel processing for the recording if it has already started.
If the file has already been decrypted the program will be processed in 'traditional' mode otherwise it will be processed in the new 'chaserun' mode.
DetectAds can also be started from a Telnet command line or from another script which could be useful if you have a batch of existing recordings to be processed.
Humax# /mod/webif/plugin/detectads/detectads -h detectads = Detect ads in recordings and bookmark or crop them detectads -d parameters = produce debug on Stdout detectads -h = produce this help detectads -pq = process queued requests (obsolete) detectads -aq recording ?-t tgtstart? = add recording to request queue tgstart = hh:mm or linux time value detectads -qq recording = query if recording on request queue detectads -dq recording = delete recording from request queue detectads -pr recording = process recording now
Note: It is not possible to specify all DetectAds options via the command line, use the Settings page to set processing options.
If the file has already been decrypted the program will be processed in 'traditional' mode otherwise it will be processed in the new 'chaserun' mode.
Options specified via the webif Settings section for DetectAds are:
- Automatically process recordings whilst recording in progress? (chaserun)
- Select the new 'chaserun' processing mode, faster but not yet widely tested
- Automatically process recordings following auto-decryption? (traditional)
- Select the slower 'traditional' automatic processing following decryption
- Folder flag, Sweeper or No automatic processing of recordings
- Select this option if you are using 'Auto Ad detection' folder flags, or Sweeper to control DetectAds or you want no automatic processing.
- Exclude channels from automatic processing
- Exclude BBC and other non commercial channels from processing, see below for more details
- Start processing recording queue
- Target time to start processing recording queue for Traditional Processing, ASAP or a time of day
- Only Bookmark ends of ad breaks?
- Select this option if you use the remote control to skip to the end of ad-breaks and do not need/want bookmarks at the start of ad-breaks. With this option you will not be able to subsequently use Crop manually to delete ad-breaks (without rerunning detection) but you can still use the automatic crop option.
- Crop recording following ad detection?
- Automatically run nicesplice to delete the adbreaks and shrink the recording file
- Delete (move to bin) original recording after processing?
- Select No if you don't trust the detection/cropping process 100% and want to keep the original around without hunting in the dustbin to find it.
- Write output recording to folder
- Choose target for writing output recording, 'Other' can only be a fixed path - if you want to vary the path use the 'Sweeper' package instead
- Limit cpu usage to
- Option will only be shown if the cpulimit package has been installed and will limit the cpu usage of the ffmpeg program (cpu hog) to approximately the specified value. This will slow down traditional processing significantly if the percentage is less than 50%, the impact on chaserun is less significant unless you are simultaneously recording two HD channels.
- This support is still experimental and it is not yet known if cpu limiting will reduce drop-outs and other faults - please report findings in the forums.
Channel exclusion list
You can type a generic matching rule such as 'BBC *' in the left hand or ctrl-click on specific channels and then click the ← button to transfer them to the left column. To remove entries from the left column position the cursor and use delete or backspace keys.
If you record from CBeebies, CBBC, etc I suggest you add them to the exclusion list since they are not covered by the default configuration.
Options specified via the webif Settings section for ChaseGet are:
- Hide ChaseGet on status display?
- Hide/Show ChaseGet activity on the webif status display
- Mute sound on ChaseGet wake from standby?
- Muting the sound prevents unwanted disturbance if the TV is on (or auto switched on) when the system awakes from standby outside peak hours.
N.B. Do not use this option if your remote is programmed to only control TV sound and can't be used to send volume controls to the Humax
- Don't mute or return to standby between and
- Use this option to specify your peak viewing times to prevent the box returning to standby without warning while it is possible that family members may be watching.
To prevent ChaseGet ever returning the system to standby specify range as 00:00 to 24:00
To allow ChaseGet to return system to standby at any time specify the range as 00:00 to 00:00
You can watch the cropped output from the chaserun process whilst recording is still in progress either through the normal TV user interface or via file sharing using an application such as VLC however there are a number of things to be born in mind.
- You can't remotely access the file via URL, only via file file sharing, there are two reasons for this.
- The output files have not been indexed by the DLNA server and
- the DLNA server only returns the amount of program known at the start of the request and so wouldn't return the entire program
- If you use the skip/fast forward buttons you are liable to run past the end as known by the Humax and playback will terminate prematurely
- If watching the -crop cropped output you have to take into account the total amount of advert breaks to be removed and the lag in producing cropped output. e.g. if you starting watching a program with 15 minutes of ads 10 minutes after it starts you will inevitably catch up with what has been processed before the end of program and playback will end. It would be safer to wait 30 minutes before attempting to start Chase play.
- The full recording length will not be shown in the program information on the TV or via webif. This is because the Humax updates the resume time during playback overwriting the actual recording length in the process.
- File will not be renamed to the original file name when recording finishes.
In general you will get the best Chase play experience viewing the -crop output on TV, starting half way or later through the recording time and leaving the remote control alone once play has started!
DetectAds works by detecting the short periods of silence that occur between ads and looks for clusters of these silence to distinguish between ad breaks and natural silences occurring during programmes.
Unfortunately sponsor bumper messages, viewer competitions, news bulletins and some trailers don't always have the necessary silences to allow them to be detected and removed.
Some ad breaks, especially in overnight programs, are too short to be reliably detected.
Ads in radio transmissions are not detected, DetectAds automatic processing will exclude recordings that have the Radio flag set.
DetectAds writes a few lines to the recmon.log depending on operating mode but the vast majority of output is written to /mod/tmp/auto.log for queued requests and to /mod/tmp/detectads.log for programmes processed during recording, records from chaseget are written to /mod/tmp/chaseget.log
Logs can be examined using the log viewer on the webif Diagnostics page or a text editor.
The amount of log data written is controlled by the "Auto-processing log level" setting in General Settings on the webif Settings page or specifying the -d option when using the command line interface.
If you experience a power cut or other failure during chaserun processing you may be left with orphaned -dec and -inp files in the /mod/tmp/ directory. These should not cause problems and will automatically be deleted.
Under the covers
In chaserun mode DetectAds sets up a pipeline where the stdout output from one program is fed into the stdin input of the next allowing them to run is parallel reducing overall elapsed time.
The stages in the chaserun pipeline are:
- This is a newly written jim program to retrieve and decrypt the input recording. It uses the system dlnahelper function to fool the DLNA server into allowing unindexed recordings to be retrieved. Curl is used to retrieve the recording in chunks with the pause between chunks allowing multiple instances of DetectAds when there are simultaneous recordings. Since the DLNA server is not active in half-awake recording state chaseget uses the IR package to force the system fully awake if needed.
- A standard linux utility is used to write the decrypted recording to the -dec file which is the input to the nsplice stage if cropping or the DetectAds output if not cropping.
- A general purpose audio and video stream processor. Extracts the audio component from the recording.
- Originally created for mythtv it detects commercial breaks by looking for periods of silence and then finding clusters of silences, the algorithm is described on the Myth TV Commercial_detection_with_silences page. Only minor changes have been made to sharpen the detection of silences and provide periodic heartbeat messages to help with cropping in the pipeline.
- Part of the DetectAds jim code analyzes the messages written by silence, updates the bookmarks in the hmt the input & -dec files, and builds cut commands for nicesplice
- (optional)This is a version of the nicesplice program modified to accept commands from stdin instead of the command line and to make the output recording visible as it is processed to permit chase playing. The .nts for the -dec file input to splice is a hard link to the .nts for the input recording.
The traditional version of DetectAds has a similar flow except that chaseget & tee are omitted, ffmpeg reads the already decrypted input file directly, The other major difference is that nsplice runs after the pipeline has completed rather than being a pipeline stage.
|2017-01-30||detectads 0.2.4-1||mymsman|| Significant update
|mymsman|| Minor update
|mymsman|| Medium update
|mymsman|| Medium update
|2016-03-20||detectads 0.2.2-7||mymsman|| Minor update
|2015-11-30||detectads 0.2.2-6||mymsman|| Minor update
|2015-11-23||detectads 0.2.2-5||mymsman|| Minor update
|2015-11-15||detectads 0.2.2-3||mymsman|| Minor update
|2015-10-28||detectads 0.2.2-1||mymsman|| Experimental update
|mymsman|| Significant update
|2015-09-30||detectads 0.2.1-6||mymsman|| Minor update
|2015-09-26||detectads 0.2.1-4||mymsman|| Bug fix
|2015-09-25||detectads 0.2.1-3||mymsman|| Bug fix
|2015-09-23||detectads 0.2.1-2||mymsman|| Bug fix
|2015-09-17||detectads 0.2.1-1||mymsman|| Minor update
|2015-09-16||detectads 0.2.1-0||mymsman|| Significant update
|2015-09-01||detectads 0.2.0-3||mymsman||Minor update
|mymsman|| Released on package catalogue
|2015-08-01||detectads 0.2.0-0||mymsman|| Significant update including:
|2014-07-04||detectads 0.0.2.2||njm||Initial version including while you wait webif and process all automation option|
Possible future enhancements - please suggest more
- User interface to change Silence setting, possibly by channel
- Replace use of ffmpeg for audio extraction
- For having the idea of bringing the mythTV ad detection to the Humax and for creating the first version of DetectAds
- For producing the Silence ad detecting package Myth TV Commercial_detection_with_silences
- For coming up with the idea of using symbolic links to allow DLNA access to files that have not yet been DLNA indexed
- For providing the recmon start recording hooks and the webif framework, adding the Addetection flag and Sweeper hooks.
- Automatic inversion of bookmarks, libsndfile improvements
- For creating nicesplice package and permitting me to update the source.
- everbody else
- Who has assisted through the forums