Always Block Google from Accessing your Site’s Search Results

If you are using Google Custom Search or another site search service on your website, make sure that the search results pages – like the one available here – are not accessible to Googlebot. This is necessary else spam domains can create serious problems for your website for no fault of yours.

Few days ago, I got an automatically generated email from Google Webmaster Tools saying that Googlebot is having trouble indexing my website labnol.org as it found a large number of new URLs. The message said:

Googlebot encountered extremely large numbers of links on your site. This may indicate a problem with your site’s URL structure… As a result Googlebot may consume much more bandwidth than necessary, or may be unable to completely index all of the content on your site.

This was a worrying signal because it meant that tons of new pages have been added to the website without my knowledge. I logged into Webmaster Tools and, as expected, there were thousands of pages that were in the crawling queue of Google.

Here’s what happened.

Some spam domains had suddenly started linking to the search page of my website using search queries in Chinese language that obviously returned no search results. Each search link is technically considered a separate web page – as they they have unique addresses – and hence the Googlebot was trying to crawl them all thinking they are different pages.

External Domains blocked with robots.txt

Because thousands of such fake links were generated in a short span of time, Googlebot assumed that these many pages have been suddenly added to the site and hence a warning message was flagged.

There are two solutions to the problem.

I can either get Google to not crawl links found on spam domains, something which is obviously not possible, or I can prevent the Googlebot from indexing these non-existent search pages on my website. The latter is possible so I fired up my VIM editor, opened the robots.txt file and added this line at the top. You’ll find this file in the root folder of your website.

User-agent: *
Disallow: /?s=*

Block Search pages from Google with robots.txt

The directive essentially prevents Googlebot, and any other search engine bot, from indexing links that have the “s” parameter the URL query string. If your site uses “q” or “search” or something else for the search variable, you may have to replace “s” with that variable.

The other option is to add the NOINDEX meta tag but that won’t have been an effective solution as Google would still have to crawl the page before deciding not to index it. Also, this is a WordPress specific issue because the Blogger robots.txt already blocks search engines from crawling the results pages.

Related: CSS for Google Custom Search

Soft 404 Errors - Google Webmaster Tools


The story, Always Block Google from Accessing your Site’s Search Results, was originally published at Digital Inspiration by Amit Agarwal on 15/04/2015 under SEO, WordPress, Internet.

Host your Podcasts on Google Drive for Free

If you are looking to publish your own audio or video podcasts, you’ll need to rent space on a public web server to host the MP3 or MP4 files of your podcast. When someone subscribes to your podcast feed in iTunes, or another podcasting app, the podcast media files will download from this server to the user’s computer or mobile phone.

Where do you host the podcast files? If you have signed up for a web hosting account, you can use the rented space to host the podcast files else you may consider using Google Drive – it is free, you can host both audio and video podcast files and there are no known bandwidth restrictions.

Google Drive for Podcast Hosting

Free Podcast Hosting on Google Drive

Google Drive offers web hosting and you can make use of this feature to host to host your own podcast show in two minutes.

Essentially, what we will do is create a new folder in Google Drive to store the podcast files and then make this folder public so anyone on the web can download episode files stored in this Google Drive folder. Any audio or video file that you upload to this Google Drive folder will have a public URL that you can use in your Podcast XML feed for publishing on iTunes.

  1. Click here and authorize the Google Script so that it can create a new public folder in your Google Drive for hosting the files.
  2. You’ll now be provide a link to the new Google Drive folder. Open the link and upload one or more podcast files – see sample folder.
  3. Next follow step #2 of the wizard and you should see a list of the uploaded podcast files and their public URLs like in this example. Copy-paste the file URLs in your iTunes RSS feed.

The podcasts will be served from googledrive.com.

Other than podcast episodes, you may also upload art work, logos and other image files that may be required for submitting your Podcast into the iTunes store.

Also see: How to Publish your own Podcast

Podcasts Files URLs on Google Drive

If you have created a podcast folder in Google Drive already and only need the URLs of the files for adding to your podcast RSS feed, here’s the trick. Make a note of the folder ID of podcast folder in Google Drive and add to the URL below (replace XYZ with your folder ID). Do make sure that privacy of your Google Drive folder is “anyone with a link can view” for people to be able to access your podcasts.

https://script.google.com/macros/s/AKfycbwLWQN93b13R5EBXetnDhc_NzUFg1Vyo5eAVN_0rsR_199uMYFG/exec?folderID=XYZ

The story, Host your Podcasts on Google Drive for Free, was originally published at Digital Inspiration by Amit Agarwal on 11/04/2015 under Google Drive, Podcasts, Internet.

Incredible Art Made Entirely In Google Drawings

Google Drawings is a simple Microsoft Paint like diagramming application that is commonly used for creating flowcharts and diagrams inside documents and presentations on Google Drive. It offers a few elementary tools before you dismiss Drawings as not worthy of any serious work, take a look at Joshua Pomeroy’s art and you’ll be amazed.

Google Drawings

Google Drawings in Google Drive

Google Drawing - Leonard Nimoy

Creating Art in Google Drawings

Joshua is a Michigan-based visual artist and he uses this basic Google Drawings app to create some very impressive and detailed vector portraits. You can browse through his work on Google Plus and all these images are created entirely inside Google Drawings.

If you are curious to know such incredible art was made, Joshua has uploaded a series of video tutorials on YouTube where he explains how he goes about creating these digital paintings from photographs inside Google Drive.


The story, Incredible Art Made Entirely In Google Drawings, was originally published at Digital Inspiration by Amit Agarwal on 07/04/2015 under Google Drive, Internet.

Teach yourself Touch Typing with Free Tools

Touch typing, or the art of typing with your 10 fingers, can do wonders to your productivity. Most technologically literate people know a bit of typing but if you can type without looking at the computer keyboard, you can get work done faster and gain an edge. Touch typing is also a useful skill to have for mobile and tablet users because when you perfectly know where the keys are, you can “Swype” faster.

Touch Typing

Master the Art of Touch Typing

I learned the basics of typing on an old typewriter. Two decades later, I am busy hunting a good typing software for my kid and the number of choices that are available on the web now is mind-boggling. I’ve shared my research here and, whether you are a beginner looking to learn typing or typing to improve your typing speed, this list should help you.

If you are just getting started, the BBC’s Dance Mat Typing should be your go-to resource. It is a browser-based game for kids where each lesson touches a different set of keys and there’s a game at the end of each level to test what you have learned. The lessons would require the Adobe Flash player enabled in your browser.

The next recommended resource is Typing.com, a free website where you’ll find interactive tutorials and drills to help you learn typing step-by-step. Typing Study is a similar resource but unlike other typing lesson that focus on the English keyboard, Typing Study has lessons for keyboards of most popular languages including Hindi and it requires no Flash Player either.

Typing Tutor - Offline

Tipp10 (Mac, Windows) and Typist (Mac only) are desktop software that teach typing offline and you also have the option to upload your own text – like pages of your favorite novel – for practice.

Practice Typing with Online Games

The best way to learn typing is through practice and there are a plethora of online typing games that will help.

The very-addictive Z-type and QWERTY Warriors are shooter games where the ships are assigned a word and you’ve to type as fast as you can to shoot these advancing ships. Type Racer and TyprX let you test your typing speed while playing a basic game of car racing with other users. 10fastfingers lets you practice typing the most common English words while Typing Club allow you to practice individual keys that you you need think need more trainging. Advanced users can try the Typing Karaoke to test their typing skills.

Ergonomic Keyboard for Touch TypingAlso see: Typing Shortcuts for iOS

Touch-typing is not difficult and you can become a pro if you set aside 30 minutes everyday for a week or two. Also, it is OK to make mistakes during training but always avoid looking down at the keyboard while typing.

Finally, invest in a good comfortable keyboard – I use Microsoft’s Sculpt keyboard and recommend it highly for ergonomics.


The story, Teach yourself Touch Typing with Free Tools, was originally published at Digital Inspiration by Amit Agarwal on 06/04/2015 under Learn, Internet.

A More Efficient Method for Embedding YouTube Videos

When you embed any YouTube video on your website using standard IFRAME tags, you’ll be surprised to know how much extra weight that YouTube video will add to your page. The web page has to download ~0.5 MB of extra resources (CSS, JavaScript and images) for rendering the YouTube video player and the files will download even if the visitor on your website has chosen not to watch the embedded YouTube video.

YouTube Player - Waterfall

The embedded video is making your page heavy and the visitor’s browser will also need to make multiple HTTP requests to render the video player. This increases the overall loading time of your page and thus affects the page speed score. The other drawback with the default YouTube embed code is that it isn’t responsive. If people view your website on a mobile phone, the video player would not resize itself accordingly.

Load YouTube Video Player On-Demand

Google Plus uses a clever workaround to reduce the time it takes to initially load the YouTube video player and we can incorporate a similar approach approach for our websites as well.

Instead of embedding the full Youtube video player, Google+ displays just the thumbnail images of a YouTube video and a “play” icon is placed over the video so that it looks like a video player. The following video is embedded using the same technique:

When the user hits the play button, the video thumbnail is replaced with the standard YouTube video player with autoplay set to 1 so it plays the video instantly. The extra player-specific resources are thus loaded only when the user has decided to play the embedded video and not otherwise.

The regular embed code for YouTube looks something like this. You specify the height of the player (in pixels), the width and the unique ID of the YouTube video.

<iframe width="320" height="180" 
     src="http://www.youtube.com/embed/LcIytqkbdlo">
</iframe>

Embed YouTube Videos Responsively without Increasing Load Time

The on-demand embed code for YouTube is slightly different since we are now embedding the video responsively and also because the IFRAME embed code is added only then user clicks the play button.

Copy-paste the following snippet anywhere in your web page where you would like the video to appear. Remember to replace VIDEOID with the actual ID of the YouTube video. The CSS and JavaScript codes are added to the template separately. Also, there’s no need to add the height and width parameter since the video will automatically occupy the width of the parent while the height is auto-calculated.

<div class="youtube-container">
   <div class="youtube-player" data-id="VIDEOID"></div>
</div>

You can copy-paste multiple blocks on the same page incase you need to embed multiple videos on the same page. The code will stay the same except that you need to change the VIDEOID for each of the blocks.

The JavaScript

The JavaScript function will scan your pages for embedded YouTube videos. If found, it will add the corresponding thumbnail image and also add the onclick event listener that will do the actual magic – replace the image with the actual YouTube video in autoplay mode.

<script>
(function() {
    var v = document.getElementsByClassName("youtube-player");
    for (var n = 0; n < v.length; n++) {
        var p = document.createElement("div");
        p.innerHTML = labnolThumb(v[n].dataset.id);
        p.onclick = labnolIframe;
        v[n].appendChild(p);
    }
})();

function labnolThumb(id) {
    return '<img class="youtube-thumb" src="//i.ytimg.com/vi/' + id + '/hqdefault.jpg"><div class="play-button"></div>';
}

function labnolIframe() {
    var iframe = document.createElement("iframe");
    iframe.setAttribute("src", "//www.youtube.com/embed/" + this.parentNode.dataset.id + "?autoplay=1&autohide=2&border=0&wmode=opaque&enablejsapi=1&controls=0&showinfo=0");
    iframe.setAttribute("frameborder", "0");
    iframe.setAttribute("id", "youtube-iframe");
    this.parentNode.replaceChild(iframe, this);
}
</script>

The CSS

Open the CSS file of your website and paste the following snippet. If you don’t have a separate CSS file, you can also place it before the closing head tag of your web template.

<style>
.youtube-container { display: block; margin: 20px auto; width: 100%; max-width: 600px; }
.youtube-player { display: block; width: 100%; /* assuming that the video has a 16:9 ratio */ padding-bottom: 56.25%; overflow: hidden; position: relative; width: 100%; height: 100%; cursor: hand; cursor: pointer; display: block; }
img.youtube-thumb { bottom: 0; display: block; left: 0; margin: auto; max-width: 100%; width: 100%; position: absolute; right: 0; top: 0; height: auto }
div.play-button { height: 72px; width: 72px; left: 50%; top: 50%; margin-left: -36px; margin-top: -36px; position: absolute; background: url("http://i.imgur.com/TxzC70f.png") no-repeat; }
#youtube-iframe { width: 100%; height: 100%; position: absolute; top: 0; left: 0; }
</style>

This method will reduce the size of your webpages by 300-400 KB while making your site mobile friendly. You may refer to the annotated code to understanding how on-demand embedding works.

Also see: Embed Google Maps Responsively


The story, A More Efficient Method for Embedding YouTube Videos, was originally published at Digital Inspiration by Amit Agarwal on 06/04/2015 under Embed, YouTube, Internet.

Download WhatsApp Photos via Dropbox

A friend has a book publishing business and have customers across the country who sell these books. They are old-school businesses that always relied on paper-based forms and fax for sending orders but now that they have WhatsApp on their mobile phones, they write their requirements on paper, snap an image and send it via WhatsApp. On this side, the received WhatsApp images are saved to the computer, printed and sent to the respective departments for processing.

The problem with the workflow is that there are quite a few steps involved after the WhatsApp image is received. Since WhatsApp neither offers a desktop client and nor does it allow integration with services like IFTTT or Zapier, the incoming images have to manually transferred from the phone to computer for sending to the printer. Is there a way to automate these steps? Yes.

When a photograph, or any media file, is received inside WhatsApp, it is automatically saved to the phone’s physical memory. The media is downloaded without you having to open the corresponding message (we are dealing with Android here, the iPhone version of WhatsApp may work differently).

Now that we know that the incoming image file is on the phone’s SD card, all we need is way to automatically transfer the file to our computer. That’s where Dropbox or Google Drive can help.

WhatsApp Dropbox Sync

Download WhatsApp Photos to Computer

We can watch the Media folder of WhatsApp and as soon as a new file is added to that folder, it is automatically synced with Dropbox. MetaCtrl has Android apps for Dropbox, Google Drive and Box that can help here.

  1. Open your Dropbox and create a folder, say WhatsApp.
  2. Launch the DropSync app and authorize it to access your Dropbox account.
  3. Set the Local (watch) folder as /storage/emulated/0/WhatsApp/Media/WhatsApp Images and set the Dropbox folder as the one that you created in Step #1.
  4. Set the sync method as “Upload only” so that it is not a 2-way sync and only WhatsApp images are sent to Dropbox but not the other way.
  5. Go to the app settings and change the Autosync Interval to 5 minutes (the default is an hour).

That’s it. Within minutes of you getting an image on WhatsApp, it will be sent to your Dropbox account in the cloud and since you have Dropbox running on your desktop too, the image will be downloaded on the computer as well.

Automatically Printing the WhatsApp Photos

The next step is to send these images from the Dropbox folder to your printer connected to the computer.

This isn’t difficult either. You can open this Windows Printing script on the computer and it will automatically send images from the Dropbox folder to the printer. A similar remote printing option is available for Mac and Linux as well.

Please do however note that this will download and print all WhatsApp images so you want to have a human layer that discards the non-essential ones.


The story, Download WhatsApp Photos via Dropbox, was originally published at Digital Inspiration by Amit Agarwal on 04/04/2015 under Dropbox, Google Drive, Print, WhatsApp, Software.

You Should Learn Regular Expressions

Regular Expressions, or RegEx, are used for searching patterns in text. For instance, a RegEx like iP(hone|ad|od)s? will find mentions of any iOS device in a document. Knowledge of Regular Expressions is essential for programmers but they can be a great skill to have for non-developers as well – people who use Microsoft Word or spend hours inside Google Spreadsheets.

RegEx in Microsoft Word

Why Learn Regular Expressions?

Regular Expressions are extremely powerful, and no less intimidating, but even basic understanding of RegEx will save you time and make your everyday computing tasks easier.

For instance, you can quickly find & replace text that matches complex patterns in Word or Vim. You can easily extract phone numbers and emails in spreadsheet cells using regex formulas. If you are creating a form in Google Drive, RegEx can help you define validation rules for user input. You can use RegEx in Gmail and Google Analytics too.

How do you learn Regular Expressions? Or, if you are already familiar, how do you take your RegEx skills to the next level? You will obviously learn by doing but there are some excellent tools and learning resources on the Internet that will take make your journey to knowing Regular Expressions more pleasant.

The Best RegEx Tools & Resources

Lea Verou’s presentation will give you a good overview of what Regular Expressions are and what you can do with them. Jeffrey Friedl’s book – Mastering Regular Expressions – is still the best printed reference for RegEx newbies and masters. You can explore RegexOne, an interactive Codecademy-like online tutorial for learning RegEx or go here for learning the basics of pattern matching.

Highlighting all the non-English characters

Highlighting all the non-English characters

RegExr is like a visual playground for Regular Expressions. You enter the text in one block and the RegEx in the other. As you edit the RegEx, the matching strings are highlighted in the input text. You can also hover over any character literal in the RegEx to know what it does. RegEx101 is a similar tool that also describes your RegEx in English as you write.

Regulex and RegExper are both open-source web apps that make it easy for you to understand and read Regular Expressions. You enter a RegEx and the tools will create a Railroad Diagram – for a string to match, it should be able to successfully move from left of the diagram all the way to the left along one of the available paths.

RegEx Visualizer

Windows users can download Expresso, a free program that will help beginners write both simple and complex regular expressions through a visual builder. Instead of coding the RegEx manually, you can select the components in a wizard. Reggy for Mac and RegEx Coach for Windows can also help you test regular expressions outside the browser.

Also see: How to Learn Coding

Once you understand the basics, head over to RegEx Golf or play this RegEx Game to test your skills. Like with everything else, you’ll only learn Regular Expressions by practicing and mere reading won’t be sufficient.


The story, You Should Learn Regular Expressions, was originally published at Digital Inspiration by Amit Agarwal on 03/04/2015 under Code, Internet.

Create an Apple ID without using a Credit Card

Some apps, Google Earth for example, are available as free downloads in the iTunes Apps store but you need a UK or US based Apple ID to install them on to your iPad or Phone. Similarly, some iBooks and podcasts have geo restrictions and may only be available to iTunes users who are logged in with an Apple ID for one of the available countries.

The workaround is simple. You can create multiple Apple IDs – like one for UK and another one for US Apps Store – and easily switch between them inside iTunes. So if you are signed-in from India, you can switch to the US store, login with your US based Apple ID and download the app that is otherwise not available in the Indian Apps Store.

Apple ID for US iTunes Store

There’s however a problem. When you create a new Apple ID, iTunes will require you to enter your credit card and the billing address of your card should be in that country. In other words, you need a US based credit card or PayPal account to create a Apple ID for the US iTunes Store. Apple will not let you create an Apple ID without entering valid payment information (see screenshot above).

That said, you can take an alternate not-so-obvious route in iTunes to create an Apple ID for any country without requiring a credit card. Here’s how:

Create Apple ID in iTunes without your Credit Card

  1. Launch the iTunes software on your computer and sign-out of your existing Apple ID. Choose Store in the menu and select Sign-out.
  2. Next scroll to the bottom of the iTunes page, click Change Country and select one from the list for which you need an Apple ID. Alternatively, you may click the country’s flag to switch to the iTunes store of another region.
  3. Now open the Apps Store inside iTunes, select any app that is free and click the Get button to download that App.
  4. iTunes will now prompt you to enter your Apple ID and password. Do not enter your existing Apple ID. Instead, click the Create Apple ID button, agree to the terms & conditions, enter your email address &amp password and minimum age.
  5. Proceed to the Payments screen and here you’ll see a new option that says NONE (see screenshot below). Select the None option, enter a dummy postal address and submit to create your new Apple ID that will be valid in the iTunes store of that country.

If you have kids at home, you can use this trick to create a separate Apple ID for the iPads, one that is not associated with your credit card and so they’ll never be able to make any accidental purchases.

Create Apple ID without Credit Card

An Easier Way to Create a New Apple ID

If you need another Apple ID but do not intend to use it with the iTunes store for download apps, there’s an easier way. Go to icloud.com, click the Create Apple ID link and choose a different country from the dropdown. Your Apple ID will be created instantly but if you decide to use it for downloading iTunes content later, you’d still need to supply the credit card.

You Cannot Create An Apple ID Because You Do Not Meet The Minimum Age Requirement

You need to be at least 13+ years old to create an Apple ID inside iTunes. However, if you enter an incorrect date, iTunes will refuse to create your Apple ID and no matter how many times you try the process, you will keep getting an error saying “you cannot create an Apple ID because you do not meet the minimum age requirements.”

This is most like a caching related bug in the iTune software but can be easily. Open Preferences inside iTunes, switch to the Advanced tab and choose Reset Cache. Quit iTunes and launch the Safari browser. Go to Preferences and under Privacy, choose the option “Remove all Website date” to clear the cache.

Open iTunes again, try downloading an app and it should not allow you to create an Apple ID without issues. Thanks Mahendra for the tip.


The story, Create an Apple ID without using a Credit Card, was originally published at Digital Inspiration by Amit Agarwal on 30/03/2015 under Itunes, Software.

How to Embed Facebook Videos in your Web Pages

Facebook, like YouTube, now allows to you easily embed videos on web pages outside Facebook. That means if you come across an interesting video on Facebook, you can easily put it on your blog without having to direct your audience to the Facebook website.

There are two important points to consider though. One, you should only embed videos that are public. Facebook does provide the embed code for private videos that are shared with you or your network but your audience won’t be able to play these videos on your website.

The other issue is that Facebook still uses the Adobe Flash player to embed videos when your website is viewed on a desktop. It automatically switches to the HTML5 format on mobile devices but if someone is viewing your website from a desktop or laptop, they would need the Shockwave Flash plugin enabled to view your embedded videos.

Add Facebook Videos to your Website

First, let’s find a video on Facebook using Graph Search. Go to the search box and type a search query like videos of cats liked by my friends or videos uploaded by me or videos uploaded by my friends of friends.

Embed Facebook Video

Next click the video thumbnail to open the video play in a lightbox window. Hover your mouse over Options and choose Embed Video from the drop-down. Facebook will provide a snippet of code that you can directly copy and paste in your website template or your blog post.

Please note that you need to be logged in to your Facebook account for generating the embed code.

Here’s a sample video embedded from Facebook.

Related: How to Embed Facebook Photos

The embedded videos are responsive and occupy the full width of the parent div. If you would like to constraint the video width or need to wrap text around the video, you’ll need to modify the .fb-video class as show in the snippet below:

<style>
  .fb-video { 
    
    /* Do not occupy the full width*/
    max-width: 300px !important; 
    
    /* Align the video to the right of a page and wrap the text */
    float: right;
    
    /* Add a Grey border to the video */
    border: 2px solid grey;
    
  }
</style>

Also see: How to Embed Almost Anything in your Website


The story, How to Embed Facebook Videos in your Web Pages, was originally published at Digital Inspiration by Amit Agarwal on 29/03/2015 under Embed, Facebook, Internet.

How to Embed Facebook Videos in your Web Pages

Facebook, like YouTube, now allows to you easily embed videos on web pages outside Facebook. That means if you come across an interesting video on Facebook, you can easily put it on your blog without having to direct your audience to the Facebook website.

There are two important points to consider though. One, you should only embed videos that are public. Facebook does provide the embed code for private videos that are shared with you or your network but your audience won’t be able to play these videos on your website.

The other issue is that Facebook still uses the Adobe Flash player to embed videos when your website is viewed on a desktop. It automatically switches to the HTML5 format on mobile devices but if someone is viewing your website from a desktop or laptop, they would need the Shockwave Flash plugin enabled to view your embedded videos.

Add Facebook Videos to your Website

First, let’s find a video on Facebook using Graph Search. Go to the search box and type a search query like videos of cats liked by my friends or videos uploaded by me or videos uploaded by my friends of friends.

Embed Facebook Video

Next click the video thumbnail to open the video play in a lightbox window. Hover your mouse over Options and choose Embed Video from the drop-down. Facebook will provide a snippet of code that you can directly copy and paste in your website template or your blog post.

Please note that you need to be logged in to your Facebook account for generating the embed code.

Here’s a sample video embedded from Facebook.

Related: How to Embed Facebook Photos

The embedded videos are responsive and occupy the full width of the parent div. If you would like to constraint the video width or need to wrap text around the video, you’ll need to modify the .fb-video class as show in the snippet below:

<style>
  .fb-video { 
    
    /* Do not occupy the full width*/
    max-width: 300px !important; 
    
    /* Align the video to the right of a page and wrap the text */
    float: right;
    
    /* Add a Grey border to the video */
    border: 2px solid grey;
    
  }
</style>

Also see: How to Embed Almost Anything in your Website


The story, How to Embed Facebook Videos in your Web Pages, was originally published at Digital Inspiration by Amit Agarwal on 29/03/2015 under Embed, Facebook, Internet.