DetectAds

DetectAds - Eliminate watching the commercial breaks!

Overview

If you are fed up with watching all of commercials for this, that, and the other or you are an habitual user of the fast forward/skip buttons on the remote when an ad break starts then DetectAds is the package for you!

DetectAds analyses a recording to detect the commercial breaks and places bookmarks at the start (optional) and end of each break so that you can just use the Bookmark button (arrowed in picture) to jump instantly to the next bookmark. Alternatively you can leave the remote languishing between the sofa cushions and have DetectAds physically remove the commercials and EPG data from the recording shrinking the recording size considerably.

Installation

DetectAds is in the "development and advanced packages" section of the catalogue so to see and install the package select "Show development and advanced packages?" in the "Advanced settings" section of the Settings page.

Automatic Processing

Traditionally automatic webif packages have been run sequentially on completed recordings under the control of the "auto" process which ran at fixed intervals. DetectAds can be run in this mode triggered from auto when auto-decryption completes.

Recent enhancements have improved the speed of this post-processing by triggering auto when a recording ends and allowing decryption to start before the system has DLNA indexed the recording but it still takes a significant time to sequentially run a number of processes on a recording.

DetectAds is an exceptionally slow process taking about 15 minutes to process an hour of SD recording which means that with the time to decrypt recording and perform crop there is a considerable delay before a fully processed recording is available for viewing.

DetectAds now introduces as an option a radically different processing model, with "Detect whilst recording" (also known as Chaserun) the detection process runs in parallel with the actual recording so the processing completes within a few seconds of the recording finishing. You can even,with some limitations, start to watch your program adfree whilst it is still recording.

With both processing modes you can choose to exclude certain commercial free channels such as BBC and Radio stations from processing (Ads on commercial radio stations are not recognized by DetectAds)

Detect whilst recording -Chaserun processing

Ad detection requires a decrypted recording to process so when detecting while recording is in progress there can be three copies of the recording in existence simultaneously: The encrypted input file, the decrypted intermediate file and the cropped & shrunk output file.

To disambiguate between the various files in both the webif and the standard Humax TV interface DetectAds adds the qualifier -dec to the file name and the word Decrypt to the programme title for the decrypted file and adds the qualifier -crop to the file name and the word Crop to the program title for the cropped file.

There is a small chance that there will be a failure in the retrieval of the decrypted file resulting in a file that is longer or shorter than it should be. This seems to occur if the system is DLNA indexing the recording whilst it is being retrieved. Since retrieval is usually within 30 seconds of actual recording it is a rare event and usually only affects the closing credits/ trailing ads but to warn of a potential problem 'Len Err' is added to the program title whilst a successful completion is shown by 'Done'.

The input file will be deleted (moved to dustbin) on completion if the 'Delete input' option is specified and there is no Length error. If the Crop option is selected the -dec file file will be deleted when processing is complete.

The DLNA server is not active in half-awake recording state so chaseget uses the IR package to force the system fully awake if needed but it does not attempt to turn power off when complete since it is not easy to determine if the user has started to watch the TV in the mean time.

Traditional processing

Unlike the chaserun processing there is a prerequisite for the file to decrypted first and bookmarks are added to it so there is no output file unless the Crop option is selected, for consistency with chaserun processing the output file name is suffixed with -crop and deletion of the input depends on the 'Delete input' option.

Setting the folder auto-decrypt option is required for the DetectAds process to be trigged but if you are using Flatten or Sweeper to rename/move files between folders it is recommended that the auto-decrypt option only be specified on the target folder to prevent the chance of DetectAds not finding the file because it has been moved.

Sweeper

The third method of automating DetectAds processing (new with version 0.2.1-0) is to set up Sweeper rules to select which recording should be processed by DetectAds. This allows you much finer control of the process than the other automation methods that only allow for exclusion based on channel name. Since Sweeper only processes files once they have completed recording and are not in-use it is inevitably slower than using Detect whilst recording.

When the DetectAds package is installed several new options are added to the Sweeper rules menus. These allow you to:

• Check whether the recording has be flagged as 'Ad-Detection done'
• Check whether the recording has already been queued for detection
• Add the recording to the DetectAds processing queue

To assist with setting up the rules a new sample set has been added to the pre-defined ruleset list. These are shown and explained below.

There are two rules in the set.

The first rule simply sets the Addetection flag in the -crop and -dec files produced by the previous version of DetectAds - it is not needed for DetectAds 0.2.1-0 (and subsequent) since the new flag is now automatically set during DetectAds processing. If you have files produced by DetectAds 0.2.0 I suggest you enable just this rule, run it once using the 'Run now' button to set the flag in existing files and then disable or delete the rule. I you only used detectads to create bookmarks you could add a 'Number of bookmarks > 0' condition to also flag files with bookmarks.

The second rule may need to be modified to meet your own requirements, as supplied it is a global rule (folder and sub-folders) that adds to the DetectAds processing queue recordings that, have no bookmarks, have no Addetection flag set, are not on a BBC channel, and are not already on the queue.

If you have other rules that rename or move the recording file they should precede this rule.

To crop or not to crop - that is the question

Detecting ads can never be 100% accurate with the risk of it failing to spot an ad break, incorrectly identifying part of a program as an ad break or mis-positioning the end of break markers.

In a year of using DetectAds I have seen very few detection errors with just a few failures to spot an ad break so I am happy to allow cropping to take place automatically - the original can always be rescued from the dustbin if needed (I do recommend use of Undelete package for safety)

However everybody's viewing habits differs and the success of DetectAds may vary by channel or program time so it is probably worth not using the 'Delete input' option until you are comfortable that cropping is working for you.

Invoking from Browse

The Addetection icon is now displayed with the other recording flags in the file list.

To process a single recording you can select "Detect Adverts" from the Opt+ pull down menu on the webif Browse file list. This will lead you to the main panel where there are two processing options:

Run Analysis now

This processes the file immediately with the results displayed in the processing routine but due to the length of time taken to analyse a recording it is recommended that this option not be used unless you have a lot of spare time or need to investigate a problem!

Run Analysis in background

This option adds the recording to the DetectAds processing queue and returns. The queue is processed sequentially and, if the system is restarted, resumes on restart so if you have a number of recordings to be processed they can be safely added to the queue without waiting for preceding files to finish processing.

The DetectAds panel displays a message below the file name if the file has already got the Addetection flag set but you can reprocess the file if you wish.

The panel also shows the queue of files awaiting processing, If the selected recording is already on the queue the two Run buttons are replaced by:

Remove from background queue

This removes the entry from the processing queue but it does not cancel processing for the recording if it has already started.

If the file has already been decrypted the program will be processed in 'traditional' mode otherwise it will be processed in the new 'chaserun' mode.

Command Line

DetectAds can also be started from a Telnet command line or from another script which could be useful if you have a batch of existing recordings to be processed.

Humax# /mod/webif/plugin/detectads/detectads -h
detectads = Detect ads in recordings and bookmark or crop them

detectads -d parameters = produce debug on Stdout
detectads -h            = produce this help
detectads -pq           = process queued requests
detectads -lq           = list queued requests
detectads -aq recording = add recording to request queue
detectads -qq recording = query if recording on request queue
detectads -dq recording = delete recording from request queue
detectads -pr recording = process recording now

Note: It is not possible to specify all DetectAds options via the command line, use the Settings page to set processing options.

If the file has already been decrypted the program will be processed in 'traditional' mode otherwise it will be processed in the new 'chaserun' mode.

Settings

Options specified via the webif Settings section for DetectAds are

Automatically process recordings whilst recording in progress?

Select the new 'chaserun' processing mode, faster but not yet widely tested

Automatically process recordings following auto-decryption?

Select the slower 'traditional' automatic processing following decryption

Sweeper or No automatic processing of recordings

Select this option if you are using Sweeper to control DetectAds or you want no automatic processing.

Exclude channels from automatic processing

Exclude BBC and other non commercial channels from processing, see below for more details

Only Bookmark ends of ad breaks?

Select this option if you use the remote control to skip to the end of ad-breaks and do not need/want bookmarks at the start of ad-breaks. With this option you will not be able to subsequently use Crop manually to delete ad-breaks (without rerunning detection) but you can still use the automatic crop option.

Crop recording following ad detection?

Automatically run nicesplice to delete the adbreaks and shrink the recording file

Dustbin/delete original recording after processing?

Select No if you don't trust the detection/cropping process 100% and want to keep the original around without hunting in the dustbin to find it.

Write output recording to

Choose target for writing output recording, 'Other' can only be a fixed path - if you want to vary the path use the 'Sweeper' package instead

Channel exclusion list

Channel exclusion is modelled closely (plagiarised) on the ChannelDel package. You can type a generic matching rule such as 'BBC *' in the left hand or ctrl-click on specific channels and then click the ← button to transfer them to the left column. To remove entries from the left column position the cursor and use delete or backspace keys.

If you record from CBeebies, CBBC, commercial radio stations I suggest you add them to the exclusion list since they are not covered by the default configuration.

Chase playing

You can watch the output from the chaserun process whilst recording is still in progress either through the normal TV user interface or via file sharing using an application such as VLC however there are a number of things to be born in mind.

• You can't remotely access the file via URL, only via file file sharing, there are two reasons for this.
  • The output files have not been indexed by the DLNA server and
  • the DLNA server only returns the amount of program known at the start of the request and so wouldn't return the entire program
• If you use the skip/fast forward buttons you are liable to run past the end as known by the Humax and playback will terminate prematurely
• If watching the -crop cropped output you have to take into account the total amount of advert breaks to be removed and the lag in producing cropped output. e.g. if you starting watching a program with 15 minutes of ads 10 minutes after it starts you will inevitably catch up with what has been processed before the end of program and playback will end. It would be safer to wait 30 minutes before attempting to start Chase play.
• If watching the -dec decrypted output you will only see the bookmarks that have already been detected since the Humax only reads the hmt file at program start, you could also inhibit the DetectAds process' ability to clean up at the end of recording which may leave orphan symlink files hanging around for the nts file.

In general you will get the best Chase play experience viewing the -crop output on TV, starting half way or later through the recording time and leaving the remote control alone once play has started!

Limitations

DetectAds works by detecting the short periods of silence that occur between ads and looks for clusters of these silence to distinguish between ad breaks and natural silences occurring during programmes.

Unfortunately sponsor bumper messages, viewer competitions, news bulletins and some trailers don't always have the necessary silences to allow them to be detected and removed.

Some ad breaks, especially in overnight programs, are too short to be reliably detected.

Ads in radio transmissions are not detected, if you record radio programs the stations should be added to the automatic processing exclusion list.

Debugging

DetectAds writes a few lines to the recmon.log or auto.log depending on operating mode but the vast majority of output is written to /mod/tmp/detectads.log and can be viewed by the log viewer on the webif Diagnostics page.

The amount of log data written is controlled by the "Auto-processing log level" setting in General Settings on the webif Settings page or specifying the -d option when using the command line interface.

If you experience a power cut or other failure during chaserun processing there is no automatic restart processing and you may be left with orphaned symlink files which could cause the system to go into "delinquent half-awake state" where it never properly goes into full standby mode. If you suspect that this is occurring use Telnet to explore the /media/My Video folder and subfolders with ls -l to look for stray symlinks which will look something like

 
lrwxrwxrwx  1 root root         55 Jul 27 14:14 Countdown_20150727_1409-dec.nts -> /mnt/hd2/My Video/Countdown/Countdown_20150727_1409.nts

these can be removed with commands like

rm "Countdown_20150727_1409-dec.nts"

the quotes around file names are needed when there are spaces in the name.

Under the covers

In chaserun mode DetectAds sets up a pipeline where the stdout output from one program is fed into the stdin input of the next allowing them to run is parallel reducing overall elapsed time.

The stages in the chaserun pipeline are:

chaseget

This is a newly written jim program to retrieve and decrypt the input recording. It uses the system dlnahelper function to fool the DLNA server into allowing unindexed recordings to be retrieved. Curl is used to retrieve the recording in chunks with the pause between chunks allowing multiple instances of DetectAds when there are simultaneous recordings. Since the DLNA server is not active in half-awake recording state chaseget uses the IR package to force the system fully awake if needed but it does not attempt to turn power off when complete

tee

A standard linux utility is used to write the decrypted recording to the -dec file which is the input to the nsplice stage if cropping or the DetectAds output if not cropping.

ffmpeg

A general purpose audio and video stream processor. Extracts the audio component from the recording.

silence

Originally created for mythtv it detects commercial breaks by looking for periods of silence and then finding clusters of silences, the algorithm is described on the Myth TV Commercial_detection_with_silences page. Only minor changes have been made to sharpen the detection of silences and provide periodic heartbeat messages to help with cropping in the pipeline.

detectads

Part of the DetectAds jim code analyzes the messages written by silence, updates the bookmarks in the hmt the input & -dec files, and builds cut commands for nicesplice

nsplice

(optional)This is a version of the nicesplice program modified to accept commands from stdin instead of the command line and to make the output recording visible as it is processed to permit chase playing. The .nts for the -dec file input to splice is a symlink to the .nts for the input recording.

The traditional version of DetectAds has a similar flow except that chaseget & tee are omitted, ffmpeg reads the already decrypted input file directly, The other major difference is that the standard nicesplice runs after the pipeline has completed rather than being a pipeline stage.

Change History

Date	Version	Author	Description
2014-07-04	0.0.2.2	njm	Initial version including while you wait webif and process all automation option
2015-08-01	0.2.0-0	mymsman	Significant update including: Channel exclusion Background processing from browse panel Command line Automatic cropping of breaks option Detect whilst recording Limited distribution
2015-08-31	0.2.0-2	mymsman	Released on package catalogue Chaseget moved to sepaate package .conf & .db moved to /mod/etc/ Avoid power on loop Minor clean up
2015-09-01	0.2.0-3	mymsman	Minor update Add ffmpeg and libsndfile to prereq list Correct bug if no ad-breaks found
2015-09-16	0.2.1-0	mymsman	Significant update Addetection flag in .hmt file (thanks af123) Automation via Sweeper (thanks af123) Sample Sweeper rules Show processing queue, remove entries from queue Only Bookmark ends of ad breaks option New icon Cosmetic improvements No deletion of inuse recordings
2015-09-17	0.2.1-1	mymsman	Minor update Show Addetection icon in browse file list (thanks af123)	2015-09-23	0.2.1-2	mymsman	Bug fix Fix exclusion of BBC, more diagnostics for short crop problem

Future Enhancements

Possible future enhancements - please suggest more

• User interface to change Silence setting, possibly by channel
• Improve interaction with status displays (Shows as playing on webif status and as multiple short entries in TV diary)
• Restart of chaserun processing
• Return system to standby when chaserun processing complete.

Acknowledgements

njm

For having the idea of bringing the mythTV ad detection to the Humax and for creating the first version of DetectAds

mythTV

For producing the Silence ad detecting package Myth TV Commercial_detection_with_silences

oatcake

For coming up with the idea of using symbolic links to allow DLNA access to files that have not yet been DLNA indexed

af123

For providing the recmon start recording hooks and the webif framework, adding the Addetection flag and Sweeper hooks.

drutt

For creating nicesplice package and permitting me to update the source.

everbody else

Who has assisted through the forums