Just because something did not have an original intent to be used in a certain way does not mean it should not be used in a new way if that way is innovative and provides value to the world.
Anything other than that understanding is called myopic thinking.
it sure as hell can be both. While robots.txt is not by original nature related to search engines, a means of security, Google has the power and resources to respect it for the sake of security. If you don't grasp that, not my problem.
Because its an opportunity for Google to help improve the securing of private information on the web. Since they already take proactive steps in other areas to improve security online, why not here?
For example - they proactively block sites their system detects that have malware or viruses. They don't have to. Its the responsibility of site owners to ensure their sites don't have malware or viruses baked in. Yet Google has chosen to help.
No, Google is NOT saying that. Yet their system is more than capable of keeping URLs out of the system that are listed in the robots file so there's no excuse why they, as a supposed security advocate, shouldn't honor robots.txt instructions. "Disallow" is pretty clear in its definition.
I'm not relying on it. I'm a forensic SEO consultant with a fair amount of digital security experience. What I'm saying is sites need to get their security methods right. At the same time, Google claims to be a security backstop, yet they allow those URLs into their system.
A citation for it? yeah half the search marketing industry. As an SEO audit professional I routinely encounter it. They list URLs, but beneath them, where a description of the file would go is a statement
A description for this result is not available because of this site's robots.txt – learn more.
They do not show all URLs that are blocked in the robots file, however if their (extremely flawed) system sees enough "other indicators" to countermand the robots instruction, they ignore that instruction.
"Other indicators" is most often "a link to that file somewhere on the site itself or pointing to the URL from another site.
While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information such as anchor text in links to the site, or the title from the Open Directory Project (www.dmoz.org), can appear in Google search results.
Which is complete bullshit. because while they're not actually indexing the CONTENT of the page, they're indexing the URL.
So in the case of a URL that includes variable parameters labeled with "order" or "customerID" or some-such, that opens up the can of WTF for anyone savvy enough to go snooping.
While Google is not ultimately responsible for other the administration of other sites, they have chosen to take a stand against hacked sites and malware ridden sites, going so far as to block them from search results pages.
Google claims to be on the side of security, yet they ignore the robots.txt file's disallow instructions, and not only that, but publicly display links found on a site where those links were clearly delineated as "disallow" in that file. As such, they are implicit in the breach.
Companies that have asshats who don't give a crap about security are the norm - its inexcusable. The fact they can even file such a suit, or that the police state can bring charges against people who expose such crap is frightening.
Just this morning in a cursory review of a prospective audit client's online presence, I did a Google search and discovered over 1,000 PDFs of customer invoices they blocked via robots.txt file but since Google now includes URLs of robots blocked files and slaps a "description not available due to robots instruction" that shit is wide open to anyone on the web, no hacking needed.
Companies need to be held accountable for their massive security failings and Google needs to be held accountable as well, even though that shit should have been completely blocked and behind a secure firewall.
The fact that this situation involved a couple reporters gives me little comfort in the notion that asshat companies might eventually be held accountable for causing such massive failings.
We need a comprehensive overhaul of the system, one NOT determined by congress or lobbyists. One that severely penalizes the asshats that cause the problem and rewards the ones who expose it.
Exactly! Thanks AC - One of the most important life lessons I needed to grasp was "know when you don't know WTF you're talking about or doing" and "step away from the ____" whatever the ____ is that you are clueless about.
While I am no better than anyone else, given that I am human, I am more likely to use what we all to quickly label as "common" sense.
It's called "critical thinking" and most people do NOT use it.
I don't care how far we've come in educating the masses. The masses are still ludite-stupid in overwhelming ways. I've witnessed too many people educated, guided, cajoled, led, informed, encouraged and otherwise shown why they need to stop being stupid with their computer and internet habits.
And too many of them nod their heads, spit out yes, thank you, what would I do without you and blah blah blah when they're given that assistance.
Only to fall right back into stupid-land.
Stupidity can't be extricated from the masses.
But just for fun - let's take it further - when you're at work as an employee, why should you even be allowed to spend time with personal email? Save it for your break, on your own time, on your own connected device.
Even if the White House threatens veto, it's possible Congress is going to shove this crap down our throats.
I just got an email from California Senator Diane Feinstein:
"As Chairman of the Senate Intelligence Committee, I will shortly be introducing a bill on information sharing which allows sharing of cyber intrusion data with the government with full immunity. In this way we hope to encourage a major effort between the public and private sector to share data so that cyber intrusion can be prevented."
(emphasis added by me)
Holy crap. I swear these wealthy, corporate shill people need to be thrown out on their asses. FULL IMMUNITY.
So like wouldn't a "prestegious" university have fiduciary responsibility to perform due diligence when coming to an arrangement like this?
I mean it doesn't take a rocket scientist (I am pretty sure no rocket scientists sit on the board at WMU but I could be wrong) to do some basic Googling to discover the overwhelming volume of complaints about the hack law school...
So yeah I am betting on the belief someone got paid off to make this happen.
hahaha a rogue mercenary providing hired thug company named "Academi" because it's most likely to be disassociative of their verified and testified and acknowledged criminal activity. (They've paid out almost $50 million in "fines" (read that payoffs to the fed so they could continue operating and continue getting contract work from the fed)...
Thank God. For a quick second, I actually thought "now that Prenda is on its deathbed, we have to pray that either John Steele opens up Salt Marsh LLC, with Alan Money as attorney of record, or we're going to have a whole lot of popcorn end up in the trash given how so little was needed during the "12 minutes".
Now at least, Malibu Media had the forethought to take advantage of this opportunity and I for one only want to know how fast we can get this over to Judge Wright's court calendar.
Sorry Tom. I just felt it important for me to clarify exactly what it was I was apologizing for because the TechDirt readers are sticklers for details and I respect this community.
I'm also very glad to say for the record that having an actual dialogue with you via email was quite productive and helped us reach a positive outcome all around, so thank you for that.
I apologize for the derogatory unwarranted attacks I initially posted. They were uncalled for.
I do not apologize for pointing out valid legitimate truth regarding the information I found on your web site that is extremely outdated.
I do not apologize for taking pity on you for your belief that the only way to prove yourself is to conduct an SEO contest where you can put your entire team of employees to work on a wild joke of a success chase where I personally am just one single person, and where my time is more valuable than playing stupid "my dick is bigger than your dick" games.
For the record, those "bullshit conferences" I speak at that you refer to in your attack on my reputation? Rand Fishkin, Jim Boykin and Matt Cutts routinely attend and speak at those same conferences.
Apparently you find yourself much more of an SEO expert than even those gentlemen as well.
it's called innovation
Just because something did not have an original intent to be used in a certain way does not mean it should not be used in a new way if that way is innovative and provides value to the world.
Anything other than that understanding is called myopic thinking.
crticial thinking
it sure as hell can be both. While robots.txt is not by original nature related to search engines, a means of security, Google has the power and resources to respect it for the sake of security. If you don't grasp that, not my problem.
Re: Re: Re: Re: Re: Re: Re: Re: Re: Re:
Because its an opportunity for Google to help improve the securing of private information on the web. Since they already take proactive steps in other areas to improve security online, why not here?
For example - they proactively block sites their system detects that have malware or viruses. They don't have to. Its the responsibility of site owners to ensure their sites don't have malware or viruses baked in. Yet Google has chosen to help.
This is no different.
Re: Re: Re: Re: Re: Re: Re: Re:
No, Google is NOT saying that. Yet their system is more than capable of keeping URLs out of the system that are listed in the robots file so there's no excuse why they, as a supposed security advocate, shouldn't honor robots.txt instructions. "Disallow" is pretty clear in its definition.
Re: Re: Re: Re: Re: Re:
I'm not relying on it. I'm a forensic SEO consultant with a fair amount of digital security experience. What I'm saying is sites need to get their security methods right. At the same time, Google claims to be a security backstop, yet they allow those URLs into their system.
Re: Re: Re: Re:
A citation for it? yeah half the search marketing industry. As an SEO audit professional I routinely encounter it. They list URLs, but beneath them, where a description of the file would go is a statement
They do not show all URLs that are blocked in the robots file, however if their (extremely flawed) system sees enough "other indicators" to countermand the robots instruction, they ignore that instruction.
"Other indicators" is most often "a link to that file somewhere on the site itself or pointing to the URL from another site.
The "learn more" link points to this Google answer page where it states:
Which is complete bullshit. because while they're not actually indexing the CONTENT of the page, they're indexing the URL.
So in the case of a URL that includes variable parameters labeled with "order" or "customerID" or some-such, that opens up the can of WTF for anyone savvy enough to go snooping.
Re: Re: Re:
uh complicit? explicit? expletive deleted? #FastRantTypingStrikesAgain
Re: Re:
While Google is not ultimately responsible for other the administration of other sites, they have chosen to take a stand against hacked sites and malware ridden sites, going so far as to block them from search results pages.
Google claims to be on the side of security, yet they ignore the robots.txt file's disallow instructions, and not only that, but publicly display links found on a site where those links were clearly delineated as "disallow" in that file. As such, they are implicit in the breach.
Re: Don't go out of your way to write a scraper program!
oh crap there's what appears to be a bomb over there. Fuck I better hide my eyes and not say anything.
OOTB you are the personification of troll=stupidity
(untitled comment)
Companies that have asshats who don't give a crap about security are the norm - its inexcusable. The fact they can even file such a suit, or that the police state can bring charges against people who expose such crap is frightening.
Just this morning in a cursory review of a prospective audit client's online presence, I did a Google search and discovered over 1,000 PDFs of customer invoices they blocked via robots.txt file but since Google now includes URLs of robots blocked files and slaps a "description not available due to robots instruction" that shit is wide open to anyone on the web, no hacking needed.
Companies need to be held accountable for their massive security failings and Google needs to be held accountable as well, even though that shit should have been completely blocked and behind a secure firewall.
The fact that this situation involved a couple reporters gives me little comfort in the notion that asshat companies might eventually be held accountable for causing such massive failings.
We need a comprehensive overhaul of the system, one NOT determined by congress or lobbyists. One that severely penalizes the asshats that cause the problem and rewards the ones who expose it.
Re: Re: Re: Re: Users are idiots
Exactly! Thanks AC - One of the most important life lessons I needed to grasp was "know when you don't know WTF you're talking about or doing" and "step away from the ____" whatever the ____ is that you are clueless about.
While I am no better than anyone else, given that I am human, I am more likely to use what we all to quickly label as "common" sense.
It's called "critical thinking" and most people do NOT use it.
Users are idiots
I don't care how far we've come in educating the masses. The masses are still ludite-stupid in overwhelming ways. I've witnessed too many people educated, guided, cajoled, led, informed, encouraged and otherwise shown why they need to stop being stupid with their computer and internet habits.
And too many of them nod their heads, spit out yes, thank you, what would I do without you and blah blah blah when they're given that assistance.
Only to fall right back into stupid-land.
Stupidity can't be extricated from the masses.
But just for fun - let's take it further - when you're at work as an employee, why should you even be allowed to spend time with personal email? Save it for your break, on your own time, on your own connected device.
Senator Feinstein Supports Full Immunity
Even if the White House threatens veto, it's possible Congress is going to shove this crap down our throats.
I just got an email from California Senator Diane Feinstein:
"As Chairman of the Senate Intelligence Committee, I will shortly be introducing a bill on information sharing which allows sharing of cyber intrusion data with the government with full immunity. In this way we hope to encourage a major effort between the public and private sector to share data so that cyber intrusion can be prevented."
(emphasis added by me)
Holy crap. I swear these wealthy, corporate shill people need to be thrown out on their asses. FULL IMMUNITY.
(untitled comment)
So like wouldn't a "prestegious" university have fiduciary responsibility to perform due diligence when coming to an arrangement like this?
I mean it doesn't take a rocket scientist (I am pretty sure no rocket scientists sit on the board at WMU but I could be wrong) to do some basic Googling to discover the overwhelming volume of complaints about the hack law school...
So yeah I am betting on the belief someone got paid off to make this happen.
Re:
Xe ---- Blackwater --- Academi LLC
hahaha a rogue mercenary providing hired thug company named "Academi" because it's most likely to be disassociative of their verified and testified and acknowledged criminal activity. (They've paid out almost $50 million in "fines" (read that payoffs to the fed so they could continue operating and continue getting contract work from the fed)...
:-)
Thank God
Thank God. For a quick second, I actually thought "now that Prenda is on its deathbed, we have to pray that either John Steele opens up Salt Marsh LLC, with Alan Money as attorney of record, or we're going to have a whole lot of popcorn end up in the trash given how so little was needed during the "12 minutes".
Now at least, Malibu Media had the forethought to take advantage of this opportunity and I for one only want to know how fast we can get this over to Judge Wright's court calendar.
(untitled comment)
This is a horrific ruling. I completely detest the notion that you can't resell something you've legally purchased.
Re: Alan why do you have to ruin a good apology that way?
Sorry Tom. I just felt it important for me to clarify exactly what it was I was apologizing for because the TechDirt readers are sticklers for details and I respect this community.
I'm also very glad to say for the record that having an actual dialogue with you via email was quite productive and helped us reach a positive outcome all around, so thank you for that.
Re: Re: Puffed Up Chests - All the Rage
Tom,
I apologize for the derogatory unwarranted attacks I initially posted. They were uncalled for.
I do not apologize for pointing out valid legitimate truth regarding the information I found on your web site that is extremely outdated.
I do not apologize for taking pity on you for your belief that the only way to prove yourself is to conduct an SEO contest where you can put your entire team of employees to work on a wild joke of a success chase where I personally am just one single person, and where my time is more valuable than playing stupid "my dick is bigger than your dick" games.
To Tom: Bullshit conferences
For the record, those "bullshit conferences" I speak at that you refer to in your attack on my reputation? Rand Fishkin, Jim Boykin and Matt Cutts routinely attend and speak at those same conferences.
Apparently you find yourself much more of an SEO expert than even those gentlemen as well.
Just sayin.