DetectAds

DetectAds - Eliminate watching the commercial breaks!

Overview

If you are fed up with watching all of commercials for this, that, and the other or you are an habitual user of the fast forward/skip buttons on the remote when an ad break starts then DetectAds is the package for you!

DetectAds analyses a recording to detect the commercial breaks and places bookmarks at the start (optional) and end of each break so that you can just use the Bookmark button (arrowed in picture) to jump instantly to the next bookmark. Alternatively you can leave the remote languishing between the sofa cushions and have DetectAds physically remove the commercials and EPG data from the recording shrinking the recording size considerably.

Installation

DetectAds is in the "development and advanced packages" section of the catalogue so to see and install the package select "Show development and advanced packages?" in the "Advanced settings" section of the Settings page. Chaseget, ir and some other packages used by DetectAds will be installed/upgraded automatically as needed.

Automatic Processing

Traditionally automatic webif packages have been run sequentially on completed recordings under the control of the "auto" process which ran at fixed intervals. DetectAds can be run in this mode triggered from auto when auto-decryption completes.

Recent enhancements have improved the speed of this post-processing by triggering auto when a recording ends and allowing decryption to start before the system has DLNA indexed the recording but it still takes a significant time to sequentially run a number of processes on a recording.

DetectAds is an exceptionally slow process taking about 15 minutes to process an hour of SD recording which means that with the time to decrypt recording and perform crop there is a considerable delay before a fully processed recording is available for viewing.

DetectAds now introduces as an option a radically different processing model, with "Detect whilst recording" (also known as Chaserun) the detection process runs in parallel with the actual recording so the processing completes within a few seconds of the recording finishing. You can even,with some limitations, start to watch your program adfree whilst it is still recording.

With both processing modes you can choose to exclude certain commercial free channels such as BBC and Radio stations from processing (Ads on radio stations are not recognized by DetectAds)

Detect whilst recording -Chaserun processing

Ad detection requires a decrypted recording to process so when detecting while recording is in progress there can be three copies of the recording in existence simultaneously: The encrypted input file, the decrypted intermediate file and the cropped & shrunk output file.

To disambiguate between the various files in both the webif and the standard Humax TV interface DetectAds adds the qualifier -dec to the file name and the word Decrypt to the programme title for the decrypted file and adds the qualifier -crop to the file name and the word Crop to the program title for the cropped file.

At the end of recording the input files will normally be deleted and the output renamed to the original file name and title. File renaming will not take place if any of the following is true:

• The "Dustbin/delete original recording after processing?" option is set to NO
• The Input or Output files are Inuse (e.g. being played or processed by another task)
• There is a length error detected (see below)

There is a small chance that there will be a failure in the retrieval of the decrypted file resulting in a file that is longer or shorter than it should be. This seems to occur if the system is DLNA indexing the recording whilst it is being retrieved. Since retrieval is usually within 30 seconds of actual recording it is a rare event and usually only affects the closing credits/ trailing ads. If it does occur the input recording will be left in situ and not replaced by the -dec (which will be deleted) or -crop files (which will be kept, If the length error is significant, more than 30 seconds long, the input file will be automatically queued to retry ad detection since it is unlikely to recur again.

It is also possible that a poor quality recording due to low strength or transmission glitches will cause the detection process to drop frames resulting in an apparently shorter recording leading to misplaced bookmarks and cropping.

If either form of length error is detected a system notification message is generated to warn you that bookmarks and cropping maybe unreliable, the input file is not deleted and -Len err is appended to the title of the -crop file (if auto cropping is active)

The DLNA server is not active in half-awake recording state so ChaseGet uses the IR package to force the system fully awake if needed. After the recording ChaseGet will now (chaseget version 0.1.1_1) attempt to turn power off when it it safe to do so but it is not easy to determine if the user has started to watch the TV in the mean time without pressing any buttons on the remote.

To reduce the risk of upsetting important family members when the box shuts down without warning (The Customised Firmware has no ability to put a 2 minute standby warning on screen like the Humax does) you can specify a time range in the ChaseGet Settings during which automatic return to standby won't operate.

You can also choose in the settings whether to Mute the sound when ChaseGet does wake the system from standby outside the peak time-range. This would prevent a sudden blast of sound if the system wakes up in the middle of the night for a scheduled recording and the TV has been left on or auto-switches on HDMI activity. A user un-muting the sound also indicates that someone is watching the TV and so ChaseGet would not then return the system to standby.

ChaseGet will return the system to StandBy if during scheduled checks which run every 20 minute all of the following are true.

• The system was awakened from Standby by ChaseGet
• The time is not within the exclusion time range
• The system isn't currently in standby
• No significant programs are running (including auto, detectads etc)- full list is in /mod/etc/chaseget.conf and can be extended if needed
• There are no active PuTTY / Telnet psuedo-terminals
• The system hasn't been been rebooted since wakeup
• No recording is being played (on set or via remote access)
• There is no active Reminder event
• No remote control buttons have been pressed since wakeup

Traditional processing

Unlike the chaserun processing there is a prerequisite for the file to decrypted first and bookmarks are added to it so there is no output file unless the Crop option is selected, for consistency with chaserun processing deletion of the input depends on the 'Delete input' option. If the input file is not deleted then the output file name is suffixed with -crop.

It is possible that a poor quality recording due to low strength or transmission glitches will cause the detection process to drop frames resulting in an apparently shorter recording leading to misplaced bookmarks. If a significant length error (>5 seconds) is detected a system notification message is generated to warn you that bookmarks maybe unreliable, the recording will not be cropped automatically.

Setting the folder auto-decrypt option is required for the DetectAds process to be trigged but if you are using Flatten or Sweeper to rename/move files between folders it is recommended that the auto-decrypt option only be specified on the target folder to prevent the chance of DetectAds not finding the file because it has been moved.

Sweeper

The third method of automating DetectAds processing (new with version 0.2.1-0) is to set up Sweeper rules to select which recording should be processed by DetectAds. This allows you much finer control of the process than the other automation methods that only allow for exclusion based on channel name. Since Sweeper only processes files once they have completed recording and are not in-use it is inevitably slower than using Detect whilst recording.

When the DetectAds package is installed several new options are added to the Sweeper rules menus. These allow you to:

• Check whether the recording has be flagged as 'Ad-Detection done'
• Check whether the recording has already been queued for detection
• Add the recording to the DetectAds processing queue

To assist with setting up the rules a new sample set has been added to the pre-defined ruleset list. These are shown and explained below.

There are two rules in the set.

The first rule simply sets the Addetection flag in the -crop and -dec files produced by the previous version of DetectAds - it is not needed for DetectAds 0.2.1-0 (and subsequent) since the new flag is now automatically set during DetectAds processing. If you have files produced by DetectAds 0.2.0 I suggest you enable just this rule, run it once using the 'Run now' button to set the flag in existing files and then disable or delete the rule. I you only used detectads to create bookmarks you could add a 'Number of bookmarks > 0' condition to also flag files with bookmarks.

The second rule may need to be modified to meet your own requirements, as supplied it is a global rule (folder and sub-folders) that adds to the DetectAds processing queue recordings that, have no bookmarks, have no Addetection flag set, are not on a BBC channel, and are not already on the queue.

If you have other rules that rename or move the recording file they should precede this rule.

To crop or not to crop - that is the question

Detecting ads can never be 100% accurate with the risk of it failing to spot an ad break, incorrectly identifying part of a program as an ad break or mis-positioning the end of break markers.

In a year of using DetectAds I have seen very few detection errors with just a few failures to spot an ad break so I am happy to allow cropping to take place automatically - the original can always be rescued from the dustbin if needed (I do recommend use of Undelete package for safety)

However everybody's viewing habits differs and the success of DetectAds may vary by channel or program time so it is probably worth not using the 'Delete input' option until you are comfortable that cropping is working for you.

If you do not use auto crop the Bookmarks set by DetectAds will now be automatically 'inverted' to meet nicesplice's requirements removing the need to perform that operation on the Crop menu page before performing the Crop operation manually.

Note: Do not use the "Only Bookmark ends of ad breaks?" if you want to retain the ability to run a manual crop,

Invoking from Browse

The Addetection icon is now displayed with the other recording flags in the file list.

To process a single recording you can select "Detect Adverts" from the Opt+ pull down menu on the webif Browse file list. This will lead you to the main panel where there are two processing options:

Run Analysis now

This processes the file immediately with the results displayed in the processing routine but due to the length of time taken to analyse a recording it is recommended that this option not be used unless you have a lot of spare time or need to investigate a problem!

Run Analysis in background

This option adds the recording to the DetectAds processing queue and returns. The queue is processed sequentially and, if the system is restarted, resumes on restart so if you have a number of recordings to be processed they can be safely added to the queue without waiting for preceding files to finish processing.

The DetectAds panel displays a message below the file name if the file has already got the Addetection flag set but you can reprocess the file if you wish.

The panel also shows the queue of files awaiting processing, If the selected recording is already on the queue the two Run buttons are replaced by:

Remove from background queue

This removes the entry from the processing queue but it does not cancel processing for the recording if it has already started.

If the file has already been decrypted the program will be processed in 'traditional' mode otherwise it will be processed in the new 'chaserun' mode.

Command Line

DetectAds can also be started from a Telnet command line or from another script which could be useful if you have a batch of existing recordings to be processed.

Humax# /mod/webif/plugin/detectads/detectads -h
detectads = Detect ads in recordings and bookmark or crop them

detectads -d parameters = produce debug on Stdout
detectads -h            = produce this help
detectads -pq           = process queued requests
detectads -lq           = list queued requests
detectads -aq recording = add recording to request queue
detectads -qq recording = query if recording on request queue
detectads -dq recording = delete recording from request queue
detectads -pr recording = process recording now

Note: It is not possible to specify all DetectAds options via the command line, use the Settings page to set processing options.

If the file has already been decrypted the program will be processed in 'traditional' mode otherwise it will be processed in the new 'chaserun' mode.

Settings

Options specified via the webif Settings section for DetectAds are:

Automatically process recordings whilst recording in progress?

Select the new 'chaserun' processing mode, faster but not yet widely tested

Automatically process recordings following auto-decryption?

Select the slower 'traditional' automatic processing following decryption

Sweeper or No automatic processing of recordings

Select this option if you are using Sweeper to control DetectAds or you want no automatic processing.

Exclude channels from automatic processing

Exclude BBC and other non commercial channels from processing, see below for more details

Only Bookmark ends of ad breaks?

Select this option if you use the remote control to skip to the end of ad-breaks and do not need/want bookmarks at the start of ad-breaks. With this option you will not be able to subsequently use Crop manually to delete ad-breaks (without rerunning detection) but you can still use the automatic crop option.

Crop recording following ad detection?

Automatically run nicesplice to delete the adbreaks and shrink the recording file

Dustbin/delete original recording after processing?

Select No if you don't trust the detection/cropping process 100% and want to keep the original around without hunting in the dustbin to find it.

Write output recording to

Choose target for writing output recording, 'Other' can only be a fixed path - if you want to vary the path use the 'Sweeper' package instead

Channel exclusion list

Channel exclusion is modelled closely (plagiarised) on the ChannelDel package. You can type a generic matching rule such as 'BBC *' in the left hand or ctrl-click on specific channels and then click the ← button to transfer them to the left column. To remove entries from the left column position the cursor and use delete or backspace keys.

If you record from CBeebies, CBBC, etc I suggest you add them to the exclusion list since they are not covered by the default configuration.

ChaseGet settings

Options specified via the webif Settings section for ChaseGet are:

Mute sound on ChaseGet wake from standby?

Muting the sound prevents unwanted disturbance if the TV is on (or auto switched on) when the system awakes from standby outside peak hours.

N.B. Do not use this option if your remote is programmed to only control TV sound and can't be used to send volume controls to the Humax

Don't mute or return to standby between and

Use this option to specify your peak viewing times to prevent the box returning to standby without warning while it is possible that family members may be watching.

To prevent ChaseGet ever returning the system to standby specify range as 00:00 to 24:00
To allow ChaseGet to return system to standby at any time specify the range as 00:00 to 00:00

Chase playing

You can watch the cropped output from the chaserun process whilst recording is still in progress either through the normal TV user interface or via file sharing using an application such as VLC however there are a number of things to be born in mind.

• You can't remotely access the file via URL, only via file file sharing, there are two reasons for this.
  • The output files have not been indexed by the DLNA server and
  • the DLNA server only returns the amount of program known at the start of the request and so wouldn't return the entire program
• If you use the skip/fast forward buttons you are liable to run past the end as known by the Humax and playback will terminate prematurely
• If watching the -crop cropped output you have to take into account the total amount of advert breaks to be removed and the lag in producing cropped output. e.g. if you starting watching a program with 15 minutes of ads 10 minutes after it starts you will inevitably catch up with what has been processed before the end of program and playback will end. It would be safer to wait 30 minutes before attempting to start Chase play.
• The full recording length will not be shown in the program information on the TV or via webif. This is because the Humax updates the resume time during playback overwriting the actual recording length in the process.
• File will not be renamed to the original file name when recording finishes.

In general you will get the best Chase play experience viewing the -crop output on TV, starting half way or later through the recording time and leaving the remote control alone once play has started!

Limitations

DetectAds works by detecting the short periods of silence that occur between ads and looks for clusters of these silence to distinguish between ad breaks and natural silences occurring during programmes.

Unfortunately sponsor bumper messages, viewer competitions, news bulletins and some trailers don't always have the necessary silences to allow them to be detected and removed.

Some ad breaks, especially in overnight programs, are too short to be reliably detected.

Ads in radio transmissions are not detected, DetectAds automatic processing will exclude recordings that have the Radio flag set.

Debugging

DetectAds writes a few lines to the recmon.log or auto.log depending on operating mode but the vast majority of output is written to /mod/tmp/detectads.log and can be viewed by the log viewer on the webif Diagnostics page.

The amount of log data written is controlled by the "Auto-processing log level" setting in General Settings on the webif Settings page or specifying the -d option when using the command line interface.

If you experience a power cut or other failure during chaserun processing you may be left with orphaned -dec and -inp files in the /mod/tmp/ directory. These should not cause problems and can be deleted.

Under the covers

In chaserun mode DetectAds sets up a pipeline where the stdout output from one program is fed into the stdin input of the next allowing them to run is parallel reducing overall elapsed time.

The stages in the chaserun pipeline are:

chaseget

This is a newly written jim program to retrieve and decrypt the input recording. It uses the system dlnahelper function to fool the DLNA server into allowing unindexed recordings to be retrieved. Curl is used to retrieve the recording in chunks with the pause between chunks allowing multiple instances of DetectAds when there are simultaneous recordings. Since the DLNA server is not active in half-awake recording state chaseget uses the IR package to force the system fully awake if needed.

tee

A standard linux utility is used to write the decrypted recording to the -dec file which is the input to the nsplice stage if cropping or the DetectAds output if not cropping.

ffmpeg

A general purpose audio and video stream processor. Extracts the audio component from the recording.

silence

Originally created for mythtv it detects commercial breaks by looking for periods of silence and then finding clusters of silences, the algorithm is described on the Myth TV Commercial_detection_with_silences page. Only minor changes have been made to sharpen the detection of silences and provide periodic heartbeat messages to help with cropping in the pipeline.

detectads

Part of the DetectAds jim code analyzes the messages written by silence, updates the bookmarks in the hmt the input & -dec files, and builds cut commands for nicesplice

nsplice

(optional)This is a version of the nicesplice program modified to accept commands from stdin instead of the command line and to make the output recording visible as it is processed to permit chase playing. The .nts for the -dec file input to splice is a hard link to the .nts for the input recording.

The traditional version of DetectAds has a similar flow except that chaseget & tee are omitted, ffmpeg reads the already decrypted input file directly, The other major difference is that nsplice runs after the pipeline has completed rather than being a pipeline stage.

Change History

Date	Version	Author	Description
2014-07-04	detectads 0.0.2.2	njm	Initial version including while you wait webif and process all automation option
2015-08-01	detectads 0.2.0-0	mymsman	Significant update including: Channel exclusion Background processing from browse panel Command line Automatic cropping of breaks option Detect whilst recording Limited distribution
2015-08-31	detectads 0.2.0-2 chaseget 0.1.0-0	mymsman	Released on package catalogue ChaseGet moved to separate package .conf & .db moved to /mod/etc/ Avoid power on loop Minor clean up
2015-09-01	detectads 0.2.0-3	mymsman	Minor update Add ffmpeg and libsndfile to prereq list Correct bug if no ad-breaks found
2015-09-16	detectads 0.2.1-0	mymsman	Significant update Addetection flag in .hmt file (thanks af123) Automation via Sweeper (thanks af123) Sample Sweeper rules Show processing queue, remove entries from queue Only Bookmark ends of ad breaks option New icon Cosmetic improvements No deletion of inuse recordings
2015-09-17	detectads 0.2.1-1	mymsman	Minor update Show Addetection icon in browse file list (thanks af123)
2015-09-23	detectads 0.2.1-2	mymsman	Bug fix Fix exclusion of BBC, more diagnostics for short crop problem
2015-09-25	detectads 0.2.1-3	mymsman	Bug fix Fix for short crop problem
2015-09-26	detectads 0.2.1-4	mymsman	Bug fix Fix incorrect dependency
2015-09-30	detectads 0.2.1-6	mymsman	Minor update Recompile Silence to work with updated libsndfile package Use -ac 2 option on ffmpeg These changes should provide better large file support
2015-10-20	detectads 0.2.2-0 chaseget 0.1.1-1	mymsman	Significant update ChaseGet return to standby ChaseGet settings page 'Inverted' bookmarks Retain original file name/title on output file Automatic reprocess on significant length error Build -dec file in /mod/tmp/ with hard link for .nts
2015-10-28	detectads 0.2.2-1	mymsman	Experimental update Use links to input file to 'hide' from DLNA indexing
2015-11-15	detectads 0.2.2-3	mymsman	Minor update Fixes the bug that caused misplaced crops when using chaserun on HD files Detects when poor quality recording cause a significant (>5 seconds) difference between detected program length and actual program length Create System notify messages when errors are detected
2015-11-23	detectads 0.2.2-5	mymsman	Minor update Avoid too frequent re queuing on repeated decryption length errors
2015-11-30	detectads 0.2.2-6	mymsman	Minor update Automatic restart of detectads following power failures during Detect while recording Decryption length failures re-queued after recording ends Reduced max ad length to 65 seconds (was 90 sec) Heartbeat at cluster start - should improve chase playing
2016-03-20	detectads 0.2.2-7	mymsman	Minor update Bypass detection for Radio programmes with Radio flag set in hmt Remove Radio filter from default configuration. Include flexview.hook file

Future Enhancements

Possible future enhancements - please suggest more

• User interface to change Silence setting, possibly by channel

Acknowledgements

njm

For having the idea of bringing the mythTV ad detection to the Humax and for creating the first version of DetectAds

mythTV

For producing the Silence ad detecting package Myth TV Commercial_detection_with_silences

oatcake

For coming up with the idea of using symbolic links to allow DLNA access to files that have not yet been DLNA indexed

af123

For providing the recmon start recording hooks and the webif framework, adding the Addetection flag and Sweeper hooks.

xyz321

Automatic inversion of bookmarks, libsndfile improvements

drutt

For creating nicesplice package and permitting me to update the source.

everbody else

Who has assisted through the forums