CareerCup

hash the url

Mark the visited urls. Like what you would do in DFS BFS etc.

And interesting variant of the problem would be this.
Say you have a series of directed edges coming up (online algo) and you need essentially construct the corresponding graph. The only condition is that you ignore the any edge which may result in a cycle. How would you do it.

Clue: Transitive closure.

A->B->C
A->D->C
There's no loop, but your algo won't work.
Actually, there's a loop if only you meet a node which is currently on the working stack.

juicer.headrun.com
visit the above sites for more detailed info about design a web crawler

visit the above sites for more detailed info about design a web crawler

@Anonymous
"A->B->C
A->D->C
There's no loop, but your algo won't work."

In the second path (A-D-C), you wouldn't even want to go to C as it's already been visited. It's a web crawler; so by marking C as visited (during the first path), you are not only preventing loops, but ALSO preventing pages from being accessed a redundant number of times.

Mark the visited urls. Like what you would do in DFS BFS etc.

And interesting variant of the problem would be this.
Say you have a series of directed edges coming up (online algo) and you need essentially construct the corresponding graph. The only condition is that you ignore the any edge which may result in a cycle. How would you do it.

Clue: Transitive closure.

Mark the visited urls. Like what you would do in DFS BFS etc.

And interesting variant of the problem would be this.
Say you have a series of directed edges coming up (online algo) and you need essentially construct the corresponding graph. The only condition is that you ignore the any edge which may result in a cycle. How would you do it.

Clue: Transitive closure.

I apologize for several posts..each submit kept throwing errors and me got confused.