DetectAds

DetectAds - Eliminate watching the commercial breaks!

Place Holder - Page under construction

Overview

If you are fed up with watching all of commercials for this, that, and the other or you are a habitual user of the fast forward/skip buttons on the remote when an adbreak starts then DetectAds is the package for you!

DetectAds analyses recording to detect the commercial breaks and places bookmarks at the start and end of each break so that you can just use the Bookmark button (arrowed in picture) to jump instantly to the next bookmark. Alternatively you can leave the remote languishing between the sofa cushions and have DetectAds physically remove the commercials and EPG data from the recording shrinking the recording size considerably.

Automatic Processing

Traditionally automatic webif packages have been run sequentially on completed recordings under the control of the "auto" process which ran at fixed intervals. DetectAds can be run in this mode triggered from auto when auto-decryption completes.

Recent enhancements have improved the speed of this post-processing by triggering auto when a recording ends and allowing decryption to start before the system has DLNA indexed the recording but it still takes a significant time to sequentially run a number of processes on a recording.

DetectAds is an exceptionally slow process taking about 15 minutes to process an hour of SD recording which means that with the time to decrypt recording and perform crop there is a considerable delay before a fully processed recording is available for viewing.

DetectAds now introduces as an option a radically different processing model, with "Detect whilst recording" (also known as ChaseRun) the detection process runs in parallel with the actual recording so the processing completes within a few seconds of the recording finishing. You can even,with some limitations, start to watch your program adfree whilst it is still recording.

With both processing modes you can choose to exclude certain commercial free channels such as BBC and Radio stations from processing. (Ads on commercial radio stations are not recognized by DetectAds)

Invoking from Browse

To process a single recording you can select "Detect Adverts" from the Opt+ pull down menu on the webif Browse file list. This will lead you to the main panel where there are two processing options:

Run Analysis now

This processes the file immediately with the results displayed in the processing routine but due to the length of time taken to analyse a recording it is recommended that this option not be used unless you have a lot of spare time or need to investigate a problem!

Run Analysis in background

This option adds the recording to the detectads processing queue and returns. The queue is processed sequentially and, if the system is restarted, resumes on restart so if you have a number of recordings to be processed they can be safely added to the queue without waiting for preceding files to finish processing.

If the file has already been decrypted the program will be processed in 'traditional' mode otherwise it will be processed in the new 'chaserun' mode.

Command Line

Detectads can also be started from a Telnet command line or from another script which could be useful if you have a batch of existing recordings to be completed.

Humax# /mod/webif/plugin/detectads/detectads -h
detectads = Detect ads in recordings and bookmark or crop them

detectads -d parameters = produce debug on Stdout
detectads -h            = produce this help
detectads -pq           = process queued requests
detectads -aq recording = add recording to request queue
detectads -pr recording = process recording now

Note: It is not possible to specify all DetectAds options via the command line, use the Settings page to set processing options.

If the file has already been decrypted the program will be processed in 'traditional' mode otherwise it will be processed in the new 'chaserun' mode.

Settings

Options specified via the webif Settings section for DetectAds are

Automatically process recordings whilst recording in progress?

Select the new 'chaserun' processing mode, faster but not yet widely tested

Automatically process recordings following auto-decryption?

Select the slower 'traditional' automatic processing following decryption

Exclude channels from automatic processing

Exclude BBC and other non commercial channels from processing, see below for more details

Crop recording following ad detection?

Automatically run nicesplice to delete the adbreaks and shrink the recording file

Dustbin/delete original recording after processing?

Select No if you don't trust the detection/cropping process 100% and want to keep the original around without hunting in the dustbin to find it.

Write output recording to

Choose target for writing output recording, 'Other' can only be a fixed path - if you want to vary the path use the 'Sweeper' package instead

Channel exclusion list

Channel exclusion is modelled closely (plagiarised) on the ChannelDel package. You can type a generic matching rule such as 'BBC *' in the left hand or ctl-click on specific channels and then click the ← button to transfer them to the left column. To remove entries from the left column position the cursor and use delete or backspace keys.

If you record from CBeebies, CBBC, commercial radio stations I suggest you add them to the exclusion list since they are not covered by the default configuration.

Chase playing

You can watch the output from the ChaseRun process whilst recording is still in progress either through the normal TV user interface or via file sharing using an application such as VLC however there are a number of things to be born in mind.

• You can't remotely access the file via DLNA, only via file file sharing, there are two reasons for this.
  • The output files have not been indexed by the DLNA server and
  • the DLNA server only returns the amount of program known at the start of the request and so wouldn't return the entire program
• If you use the skip/fast forward buttons you are liable to run past the end as known by the Humax and playback will terminate prematurely
• If watching the -crop cropped output you have to take into account the total amount of advert breaks to be removed and the lag in producing cropped output. e.g. if you starting watching a program with 15 minutes of ads 10 minutes after it starts you will inevitably catch up with what has been processed before the end of program and playback will end. It would be safer to wait 30 minutes before attempting to start Chase play.
• If watching the -dec decrypted output you will only see the bookmarks that have already been detected since the Humax only reads the hmt file at program start, you could also inhibit the DetectAds process' ability to clean up at the end of recording which may leave orphan symlink files hanging around for the nts file.

In general you will get the best Chase play experience viewing the -crop output on TV, starting half way or later through the recording time and leaving the remote control alone once play has started!

Debugging

DetectAds writes a few lines to the recmon.log or auto.log depending on operating mode but the vast majority of output is written to /mod/tmp/detectads.log and can be viewed by the log viewer on the webif Diagnostics page.

The amount of log data written is controlled by the "Auto-processing log level" setting in General Settings on the webif Settings page or specifying the -d option when using the command line interface.

If you experience a power cut or other failure during Chaserun processing there is no automatic restart processing and you may be left with orphaned symlink files which could cause the system to go into "delinquent half-awake state" where it never properly goes into full standby mode. If you suspect that this is occurring use Telnet to explore the /media/My Video folder and subfolders with ls -l to look for stray symlinks which will look something like

 
lrwxrwxrwx  1 root root         55 Jul 27 14:14 Countdown_20150727_1409-dec.nts -> /mnt/hd2/My Video/Countdown/Countdown_20150727_1409.nts

these can be removed with commands like

rm "Countdown_20150727_1409-dec.nts"

the quotes around file names are needed when there are spaces in the name.

Under the covers

In Chaserun mode DetectAds sets up a pipeline where the stdout output from one program is fed into the stdin input of the next allowing them to run is parallel reducing overall elapsed time.

The stages in the Chaserun pipeline are:

chaseget

This is a newly written jim program to retrieve and decrypt the input recording. It uses the system dlnahelper function to fool the dlna server into allowing unindexed recordings to be retrieved. Curl is used to retrieve the recording in chunks with the pause between chunks allowing multiple instances of DetectAds when there are simultaneous recordings. Since the dlna server is not active in half-awake recording state chaseget uses the IR package to force the system fully awake if needed but it does not attempt to turn power off when complete

tee

A standard linux utility is used to write the decrypted recording to the -dec file which is the input to the nsplice stage if cropping or the detectads output if not cropping.

ffmpeg

A general purpose audio and video stream processor. Extracts the audio component from the recording.

silence

Originally created for mythtv it detects commercial breaks by looking for periods of silence and then finding clusters of silences, the algorithm is described on the Myth TV Commercial_detection_with_silences page. Only minor changes have been made to sharpen the detection of silences and provide periodic heartbeat messages to help with cropping in the pipeline.

detectads

Part of the detctads jim code analyzes the messages written by silence, updates the bookmarks in the hmt the input & -dec files, and builds cut commands for nicesplice

nsplice

(optional)This is a version of the nicesplice program modified to accept commands from stdin instead of the command line and to make the output recording visible as it is processed to permit chase playing. The .nts for the -dec file input to splice is a symlink to the .nts for the input recording.

The traditional version of detects ads has a similar flow except that chaseget & tee are omitted, ffmpeg reads the already decrypted input file directly, The other major difference is that the standard nicesplice runs after the pipeline has completed rather than being a pipeline stage.

Change History

Date	Version	Author	Description
2014-07-04	0.0.2.2	njm	Initial version including while you wait webif and process all automation option
2015-08-01	0.2.0.0	mymsman	Significant update including: Channel exclusion Background processing from browse panel Command line Automatic cropping of breaks option Detect whilst recording