DetectAds

DetectAds - Eliminate watching the commercial breaks!

Overview

If you are fed up with watching all of commercials for this, that, and the other or you are an habitual user of the fast forward/skip buttons on the remote when an ad break starts then DetectAds is the package for you!

DetectAds analyses a recording to detect the commercial breaks and places bookmarks at the start (optional) and end of each break so that you can just use the Bookmark button (arrowed in picture) to jump instantly to the next bookmark. Alternatively you can leave the remote languishing between the sofa cushions and have DetectAds physically remove the commercials and EPG data from the recording shrinking the recording size considerably.

Installation

DetectAds is in the Packages catalogue. Chaseget, ir and some other packages used by DetectAds will be installed/upgraded automatically as needed.

Automatic Processing

Traditionally automatic webif packages have been run sequentially on completed recordings under the control of the "auto" process which ran at fixed intervals. DetectAds can be run in this mode triggered from auto when auto-decryption completes.

Recent enhancements have improved the speed of this post-processing by triggering auto when a recording ends and allowing decryption to start before the system has DLNA indexed the recording but it still takes a significant time to sequentially run a number of processes on a recording.

DetectAds is an exceptionally slow process taking about 15 minutes to process an hour of SD recording which means that with the time to decrypt recording and perform crop there is a considerable delay before a fully processed recording is available for viewing.

DetectAds now introduces as an option a radically different processing model, with "Detect whilst recording" (also known as Chaserun) the detection process runs in parallel with the actual recording so the processing completes within a few seconds of the recording finishing. You can even,with some limitations, start to watch your program adfree whilst it is still recording.

With both processing modes you can choose to exclude certain commercial free channels such as BBC and Radio stations from processing (Ads on radio stations are not recognized by DetectAds)

Some users have reported that when the system is exceptionally busy such as when recording multiple HD channels and playing other recordings that running DetectAds can cause Picture and Sound breakups on the current recordings, if you experience such problems you can limit the amount of CPU used by the processor intensive part of DetectAds or / and defer processing until the system is quieter.

Detect whilst recording -Chaserun processing

Ad detection requires a decrypted recording to process so when detecting while recording is in progress there can be three copies of the recording in existence simultaneously: The encrypted input file, the decrypted intermediate file and the cropped & shrunk output file.

To disambiguate between the various files in both the webif and the standard Humax TV interface DetectAds adds the qualifier -dec to the file name and the word Decrypt to the programme title for the decrypted file and adds the qualifier -crop to the file name and the word Crop to the program title for the cropped file.

At the end of recording the input files will normally be deleted and the output renamed to the original file name and title. File renaming will not take place if any of the following is true:

• The "Dustbin/delete original recording after processing?" option is set to NO
• The Input or Output files are Inuse (e.g. being played or processed by another task)
• There is a length error detected (see below)

There is a small chance that there will be a failure in the retrieval of the decrypted file resulting in a file that is longer or shorter than it should be. This seems to occur if the system is DLNA indexing the recording whilst it is being retrieved. Since retrieval is usually within 30 seconds of actual recording it is a rare event and usually only affects the closing credits/ trailing ads. If it does occur the input recording will be left in situ and not replaced by the -dec (which will be deleted) or -crop files (which will be kept, If the length error is significant, more than 30 seconds long, the input file will be automatically queued to retry ad detection since it is unlikely to recur again.

It is also possible that a poor quality recording due to low strength or transmission glitches will cause the detection process to drop frames resulting in an apparently shorter recording leading to misplaced bookmarks and cropping.

If either form of length error is detected a system notification message is generated to warn you that bookmarks and cropping maybe unreliable, the input file is not deleted and -Len err is appended to the title of the -crop file (if auto cropping is active)

The DLNA server is not active in half-awake recording state so ChaseGet uses the IR package to force the system fully awake if needed. After the recording ChaseGet will now (chaseget version 0.1.1_1) attempt to turn power off when it it safe to do so but it is not easy to determine if the user has started to watch the TV in the mean time without pressing any buttons on the remote.

To reduce the risk of upsetting important family members when the box shuts down without warning (The Customised Firmware has no ability to put a 2 minute standby warning on screen like the Humax does) you can specify a time range in the ChaseGet Settings during which automatic return to standby won't operate.

You can also choose in the settings whether to Mute the sound when ChaseGet does wake the system from standby outside the peak time-range. This would prevent a sudden blast of sound if the system wakes up in the middle of the night for a scheduled recording and the TV has been left on or auto-switches on HDMI activity. A user un-muting the sound also indicates that someone is watching the TV and so ChaseGet would not then return the system to standby.

ChaseGet will return the system to StandBy if during scheduled checks which run every 20 minute all of the following are true.

• The system was awakened from Standby by ChaseGet
• The time is not within the exclusion time range
• The system isn't currently in standby
• No significant programs are running (including auto, detectads etc)- full list is in /mod/etc/chaseget.conf and can be extended if needed
• There are no active PuTTY / Telnet psuedo-terminals
• The system hasn't been been rebooted since wakeup
• No recording is being played (on set or via remote access)
• There is no active Reminder event
• No remote control buttons have been pressed since wakeup

Traditional processing

Unlike the chaserun processing there is a prerequisite for the file to decrypted first and bookmarks are added to it so there is no output file unless the Crop option is selected, for consistency with chaserun processing deletion of the input depends on the 'Delete input' option. If the input file is not deleted then the output file name is suffixed with -crop.

It is possible that a poor quality recording due to low strength or transmission glitches will cause the detection process to drop frames resulting in an apparently shorter recording leading to misplaced bookmarks. If a significant length error (>5 seconds) is detected a system notification message is generated to warn you that bookmarks maybe unreliable, the recording will not be cropped automatically.

Setting the folder auto-decrypt option is required for the DetectAds process to be trigged but if you are using Flatten or Sweeper to rename/move files between folders it is recommended that the auto-decrypt option only be specified on the target folder to prevent the chance of DetectAds not finding the file because it has been moved.

If you wish wish to defer DetectAds processing to a time when the system is less busy you can specify a target start time on the Setiings page, the target time is the earliest start time but processing may start later depending on when the regular DetectAds task starts and how many other items need to be processed on the queue. You must of course ensure the the system is switched on at the target processing time and remains on long enough to process all of the queued items (possibly set up a scheduled repeating reminder event)

Folder Flags

There are two new folder flags Auto DetectAds and No DetectAds

These work in conjunction with the existing Channel exclusions rules so it is not necessary to add the flag to each directory if you use the existing DetctAds Auto processing options (Automatically process whilst recording in progress? (chaserun) or Automatically process recording following auto-decryption? (traditional)

You can also use the Auto DetectAds flag to process exist recordings in a folder.

The Auto DetectAds flag means all recordings in a folder will be processed regardless of the recording channel and whether they have been decrypted. The No DetectAds flag means none of the recordings in a folder will be processed regardless of the recording channel. In the absence of a flag recordings will be processed during recording (chaserun) or following decryption (traditional) if they are on a channel that has not been excluded by the channel filters.

If you routinely want to detect ads in all commercial programmes as I do you don't need to use the new flags at all. If you only want a subset of programmes processed you can use the flags to include/exclude directories overriding the channel filters

For instance you could set No DetectAds on the 'My Videos' root folder to prevent auto ad detection of one-off recordings (I don't recommend setting Auto DetectAds on 'My Videos' since this would override the channel selection filters and cause detection of all one-off recordings including those on BBC channels)

The processing is (deliberately) different from the other auto options where you have to have a flag set before any processing happens.

Sweeper

Another method of automating DetectAds processing is to set up Sweeper rules to select which recording should be processed by DetectAds. This allows you much finer control of the process than the other automation methods that only allow for exclusion based on channel name or folder. Since Sweeper only processes files once they have completed recording and are not in-use it is inevitably slower than using Detect whilst recording.

When the DetectAds package is installed several new options are added to the Sweeper rules menus. These allow you to:

• Check whether the recording has been flagged as 'Ad-Detection done'
• Check whether the recording has already been queued for detection
• Add the recording to the auto processing queue, optionally specifying the time for processing to start,

To assist with setting up the rules a new sample set has been added to the pre-defined ruleset list. These are shown and explained below.

There is one rule in the set.

The rule may need to be modified to meet your own requirements, as supplied it is a file rule (recursive into child folders) that adds to the auto processing queue recordings that, have no bookmarks, have no Addetection flag set, are not on a BBC channel, and are not already on the queue.

If you have other rules that rename or move the recording file they should precede this rule.

If you wish wish to defer DetectAds processing to a time when the system is less busy you can specify current time conditions in the Sweeper rule to ensure recordings are only queued when you want them processed. You must of course ensure the the system is switched on at the target processing time and remains on long enough to process all of the queued items (possibly set up a scheduled repeating reminder event)

Note: Previous versions of DetectAds provided specific DetectAds queuing actions and conditions which have now been superseded by the standard 'Queue recording for' action, the older actions remain for compatibly with pre-existing rule sets. The -t hh;mm target start time is only supported with the older action.

To crop or not to crop - that is the question

Detecting ads can never be 100% accurate with the risk of it failing to spot an ad break, incorrectly identifying part of a program as an ad break or mis-positioning the end of break markers.

In over a year of using DetectAds I have seen very few detection errors with just a few failures to spot an ad break so I am happy to allow cropping to take place automatically - the original can always be rescued from the dustbin if needed (I do recommend use of Undelete package for safety)

However everybody's viewing habits differs and the success of DetectAds may vary by channel or program time so it is probably worth not using the 'Delete input' option until you are comfortable that cropping is working for you.

If you do not use auto crop the Bookmarks set by DetectAds will now be automatically 'inverted' to meet nicesplice's requirements removing the need to perform that operation on the Crop menu page before performing the Crop operation manually.

Note: Do not use the "Only Bookmark ends of ad breaks?" if you want to retain the ability to run a manual crop,

Invoking from Browse or FlexView

The Addetection icon is now displayed with the other recording flags in the file list.

To process multiple recordings you can select the recordings, choose 'Ad Detection' from the list and then click on the 'Queue For' button at the bottom of the page. Remember Ad Detection is slow so be be sparing when adding batches of existing recordings to the queue.

To process a single recording you can select "Detect Adverts" from the Opt+ pull down menu on the webif Browse file list. This will lead you to the main panel where there are two processing options:

Run Analysis now

This processes the file immediately with the results displayed in the processing routine but due to the length of time taken to analyse a recording it is recommended that this option not be used unless you have a lot of spare time or need to investigate a problem!

Run Analysis in background

This option adds the recording to the Auto processing queue and returns. The queue is processed sequentially and, if the system is restarted, resumes on restart so if you have a number of recordings to be processed they can be safely added to the queue without waiting for preceding files to finish processing.

If you wish wish to defer DetectAds processing to a time when the system is less busy you can specify a target start time or ASAP on the panel. The target time is the earliest start time but processing may start later depending on when the regular auto processing task starts and how many other items need to be processed on the queue. You must of course ensure the the system is switched on at the target processing time and remains on long enough to process all of the queued items (possibly set up a scheduled repeating reminder event)

If the selected recording is already on the queue the two Run buttons are replaced by:

Remove from background queue

This removes the entry from the processing queue but it does not cancel processing for the recording if it has already started.

If the file has already been decrypted the program will be processed in 'traditional' mode otherwise it will be processed in the new 'chaserun' mode.

Command Line

DetectAds can also be started from a Telnet command line or from another script which could be useful if you have a batch of existing recordings to be processed.

Humax# /mod/webif/plugin/detectads/detectads -h
detectads = Detect ads in recordings and bookmark or crop them

detectads -d parameters = produce debug on Stdout
detectads -h            = produce this help
detectads -pq           = process queued requests (obsolete)
detectads -aq recording ?-t tgtstart? = add recording to request queue
         tgstart = hh:mm or linux time value
detectads -qq recording = query if recording on request queue
detectads -dq recording = delete recording from request queue
detectads -pr recording = process recording now

Note: It is not possible to specify all DetectAds options via the command line, use the Settings page to set processing options.

If the file has already been decrypted the program will be processed in 'traditional' mode otherwise it will be processed in the new 'chaserun' mode.

Settings

Options specified via the webif Settings section for DetectAds are:

Automatically process recordings whilst recording in progress? (chaserun)

Select the new 'chaserun' processing mode, faster but not yet widely tested

Automatically process recordings following auto-decryption? (traditional)

Select the slower 'traditional' automatic processing following decryption

Folder flag, Sweeper or No automatic processing of recordings

Select this option if you are using 'Auto Ad detection' folder flags, or Sweeper to control DetectAds or you want no automatic processing.

Exclude channels from automatic processing

Exclude BBC and other non commercial channels from processing, see below for more details

Start processing recording queue

Target time to start processing recording queue for Traditional Processing, ASAP or a time of day

Only Bookmark ends of ad breaks?

Select this option if you use the remote control to skip to the end of ad-breaks and do not need/want bookmarks at the start of ad-breaks. With this option you will not be able to subsequently use Crop manually to delete ad-breaks (without rerunning detection) but you can still use the automatic crop option.

Crop recording following ad detection?

Automatically run nicesplice to delete the adbreaks and shrink the recording file

Delete (move to bin) original recording after processing?

Select No if you don't trust the detection/cropping process 100% and want to keep the original around without hunting in the dustbin to find it.

Write output recording to folder

Choose target for writing output recording, 'Other' can only be a fixed path - if you want to vary the path use the 'Sweeper' package instead

Limit cpu usage to

Option will only be shown if the cpulimit package has been installed and will limit the cpu usage of the ffmpeg program (cpu hog) to approximately the specified value. This will slow down traditional processing significantly if the percentage is less than 50%, the impact on chaserun is less significant unless you are simultaneously recording two HD channels.

This support is still experimental and it is not yet known if cpu limiting will reduce drop-outs and other faults - please report findings in the forums.

Channel exclusion list

You can type a generic matching rule such as 'BBC *' in the left hand or ctrl-click on specific channels and then click the ← button to transfer them to the left column. To remove entries from the left column position the cursor and use delete or backspace keys.

If you record from CBeebies, CBBC, etc I suggest you add them to the exclusion list since they are not covered by the default configuration.

ChaseGet settings

Options specified via the webif Settings section for ChaseGet are:

Hide ChaseGet on status display?

Hide/Show ChaseGet activity on the webif status display

Mute sound on ChaseGet wake from standby?

Muting the sound prevents unwanted disturbance if the TV is on (or auto switched on) when the system awakes from standby outside peak hours.

N.B. Do not use this option if your remote is programmed to only control TV sound and can't be used to send volume controls to the Humax

Don't mute or return to standby between and

Use this option to specify your peak viewing times to prevent the box returning to standby without warning while it is possible that family members may be watching.

To prevent ChaseGet ever returning the system to standby specify range as 00:00 to 24:00
To allow ChaseGet to return system to standby at any time specify the range as 00:00 to 00:00

Chase playing

You can watch the cropped output from the chaserun process whilst recording is still in progress either through the normal TV user interface or via file sharing using an application such as VLC however there are a number of things to be born in mind.

• You can't remotely access the file via URL, only via file file sharing, there are two reasons for this.
  • The output files have not been indexed by the DLNA server and
  • the DLNA server only returns the amount of program known at the start of the request and so wouldn't return the entire program
• If you use the skip/fast forward buttons you are liable to run past the end as known by the Humax and playback will terminate prematurely
• If watching the -crop cropped output you have to take into account the total amount of advert breaks to be removed and the lag in producing cropped output. e.g. if you starting watching a program with 15 minutes of ads 10 minutes after it starts you will inevitably catch up with what has been processed before the end of program and playback will end. It would be safer to wait 30 minutes before attempting to start Chase play.
• The full recording length will not be shown in the program information on the TV or via webif. This is because the Humax updates the resume time during playback overwriting the actual recording length in the process.
• File will not be renamed to the original file name when recording finishes.

In general you will get the best Chase play experience viewing the -crop output on TV, starting half way or later through the recording time and leaving the remote control alone once play has started!

Limitations

DetectAds works by detecting the short periods of silence that occur between ads and looks for clusters of these silence to distinguish between ad breaks and natural silences occurring during programmes.

Unfortunately sponsor bumper messages, viewer competitions, news bulletins and some trailers don't always have the necessary silences to allow them to be detected and removed.

Some ad breaks, especially in overnight programs, are too short to be reliably detected.

Ads in radio transmissions are not detected, DetectAds automatic processing will exclude recordings that have the Radio flag set.

Debugging

DetectAds writes a few lines to the recmon.log depending on operating mode but the vast majority of output is written to /mod/tmp/auto.log for queued requests and to /mod/tmp/detectads.log for programmes processed during recording, records from chaseget are written to /mod/tmp/chaseget.log

Logs can be examined using the log viewer on the webif Diagnostics page or a text editor.

The amount of log data written is controlled by the "Auto-processing log level" setting in General Settings on the webif Settings page or specifying the -d option when using the command line interface.

If you experience a power cut or other failure during chaserun processing you may be left with orphaned -dec and -inp files in the /mod/tmp/ directory. These should not cause problems and will automatically be deleted.

Under the covers

In chaserun mode DetectAds sets up a pipeline where the stdout output from one program is fed into the stdin input of the next allowing them to run is parallel reducing overall elapsed time.

The stages in the chaserun pipeline are:

chaseget

This is a newly written jim program to retrieve and decrypt the input recording. It uses the system dlnahelper function to fool the DLNA server into allowing unindexed recordings to be retrieved. Curl is used to retrieve the recording in chunks with the pause between chunks allowing multiple instances of DetectAds when there are simultaneous recordings. Since the DLNA server is not active in half-awake recording state chaseget uses the IR package to force the system fully awake if needed.

tee

A standard linux utility is used to write the decrypted recording to the -dec file which is the input to the nsplice stage if cropping or the DetectAds output if not cropping.

ffmpeg

A general purpose audio and video stream processor. Extracts the audio component from the recording.

silence

Originally created for mythtv it detects commercial breaks by looking for periods of silence and then finding clusters of silences, the algorithm is described on the Myth TV Commercial_detection_with_silences page. Only minor changes have been made to sharpen the detection of silences and provide periodic heartbeat messages to help with cropping in the pipeline.

detectads

Part of the DetectAds jim code analyzes the messages written by silence, updates the bookmarks in the hmt the input & -dec files, and builds cut commands for nicesplice

nsplice

(optional)This is a version of the nicesplice program modified to accept commands from stdin instead of the command line and to make the output recording visible as it is processed to permit chase playing. The .nts for the -dec file input to splice is a hard link to the .nts for the input recording.

The traditional version of DetectAds has a similar flow except that chaseget & tee are omitted, ffmpeg reads the already decrypted input file directly, The other major difference is that nsplice runs after the pipeline has completed rather than being a pipeline stage.

Change History

Date	Version	Author	Description
2017-01-30	detectads 0.2.4-1	mymsman	Significant update It now uses the system queue for all deferred requests rather than maintaining its own database The old queue will be processed if there are outstanding requests on it but no new entries will be added and it will eventually be deleted. You can add Ad Detection requests to the queue using the Browse and Sweeper 'Queue for' options The old DetectAds Sweeper actions remain for compatibility with existing rules but now use the system queue The sample Sweeper rule has been updated to use the 'Queue recording for' action There are two new folder flags Auto DetectAds and No DetectAds These work in conjunction with the existing Channel exclusions rules so it is not necessary to add the flag to each directory if you use the existing DetctAds Auto processing options (Automatically process whilst recording in progress? (chaserun) or Automatically process recording following auto-decryption? (traditional) ) You can use these flags to include additional directories (Auto DetectAds) where you want all (including pre-existing recordings) ad detected or exclude directories (No DetectAds) which would otherwise be subject to ad detection based on channel filters. For instance you could set No DetectAds on the 'My Videos' root folder to prevent auto ad detection of one-off recordings (I don't recommend setting Auto DetectAds on 'My Videos' since this would override the channel selection filters and cause detection of all one-off recordings including those on BBC channels) DetectAds processing can still be deferred to, for example overnight, by setting a target start time on the settings page (remember to leave machine on or set a reminder) Logging for Queued requests will now be in auto.log but remain in detectads.log for detect while recording. Chaseget logging will be in chaseget.log. Chaserun processing remains a process while recording operation though it does create a temporary queue entry as a safeguard against failures during recording.
2016-11-16	detectads 0.2.3-3 chaseget 0.1.1-6	mymsman	Minor update Show DetectAds and ChaseGet activity in webif status display Option on settings menu to hide ChaseGet status display
2016-05-25	detectads 0.2.3-2 chaseget 0.1.1-5	mymsman	Medium update New option to specify target start time for queued requests: On Settings page On Browse detectads page As sweeper parameter -t hh:mm via command line -t hh:mm or -t timevalue Minor cleanup of settings page Sweeper rules restored to Global Chaseget now uses system uptime to determine boot time
2016-03-28	detectads 0.2.3-0 chaseget 0.1.1-2	mymsman	Medium update If cpulimit package is installed: New settings option to specify limit Limits ffmpeg usage to specified value Automatic clean up of orphaned files Detect and log 'pipe broken' in chaseget 'Back to media browser' should work in flexview Sweeper rules no longer Global
2016-03-20	detectads 0.2.2-7	mymsman	Minor update Bypass detection for Radio programmes with Radio flag set in hmt Remove Radio filter from default configuration. Include flexview.hook file
2015-11-30	detectads 0.2.2-6	mymsman	Minor update Automatic restart of detectads following power failures during Detect while recording Decryption length failures re-queued after recording ends Reduced max ad length to 65 seconds (was 90 sec) Heartbeat at cluster start - should improve chase playing
2015-11-23	detectads 0.2.2-5	mymsman	Minor update Avoid too frequent re queuing on repeated decryption length errors
2015-11-15	detectads 0.2.2-3	mymsman	Minor update Fixes the bug that caused misplaced crops when using chaserun on HD files Detects when poor quality recording cause a significant (>5 seconds) difference between detected program length and actual program length Create System notify messages when errors are detected
2015-10-28	detectads 0.2.2-1	mymsman	Experimental update Use links to input file to 'hide' from DLNA indexing
2015-10-20	detectads 0.2.2-0 chaseget 0.1.1-1	mymsman	Significant update ChaseGet return to standby ChaseGet settings page 'Inverted' bookmarks Retain original file name/title on output file Automatic reprocess on significant length error Build -dec file in /mod/tmp/ with hard link for .nts
2015-09-30	detectads 0.2.1-6	mymsman	Minor update Recompile Silence to work with updated libsndfile package Use -ac 2 option on ffmpeg These changes should provide better large file support
2015-09-26	detectads 0.2.1-4	mymsman	Bug fix Fix incorrect dependency
2015-09-25	detectads 0.2.1-3	mymsman	Bug fix Fix for short crop problem
2015-09-23	detectads 0.2.1-2	mymsman	Bug fix Fix exclusion of BBC, more diagnostics for short crop problem
2015-09-17	detectads 0.2.1-1	mymsman	Minor update Show Addetection icon in browse file list (thanks af123)
2015-09-16	detectads 0.2.1-0	mymsman	Significant update Addetection flag in .hmt file (thanks af123) Automation via Sweeper (thanks af123) Sample Sweeper rules Show processing queue, remove entries from queue Only Bookmark ends of ad breaks option New icon Cosmetic improvements No deletion of inuse recordings
2015-09-01	detectads 0.2.0-3	mymsman	Minor update Add ffmpeg and libsndfile to prereq list Correct bug if no ad-breaks found
2015-08-31	detectads 0.2.0-2 chaseget 0.1.0-0	mymsman	Released on package catalogue ChaseGet moved to separate package .conf & .db moved to /mod/etc/ Avoid power on loop Minor clean up
2015-08-01	detectads 0.2.0-0	mymsman	Significant update including: Channel exclusion Background processing from browse panel Command line Automatic cropping of breaks option Detect whilst recording Limited distribution
2014-07-04	detectads 0.0.2.2	njm	Initial version including while you wait webif and process all automation option

Future Enhancements

Possible future enhancements - please suggest more

• User interface to change Silence setting, possibly by channel
• Replace use of ffmpeg for audio extraction

Acknowledgements

njm

For having the idea of bringing the mythTV ad detection to the Humax and for creating the first version of DetectAds

mythTV

For producing the Silence ad detecting package Myth TV Commercial_detection_with_silences

oatcake

For coming up with the idea of using symbolic links to allow DLNA access to files that have not yet been DLNA indexed

af123

For providing the recmon start recording hooks and the webif framework, adding the Addetection flag and Sweeper hooks.

xyz321

Automatic inversion of bookmarks, libsndfile improvements

drutt

For creating nicesplice package and permitting me to update the source.

everbody else

Who has assisted through the forums