Invalid Argument in Open method for web scraping
I am trying to scrape some data from the ancestry, I have a .net background but thought i'd try a bit of python for a project.
I'm falling at the first step, Firstly i am trying to open this page and then just print out the rows.
from requests import get
from requests.exceptions import RequestException
from contextlib import closing
from bs4 import BeautifulSoup
raw_html = open('https://www.ancestry.co.uk/search/collections/britisharmyservice/?
birth=_merthyr+tydfil-wales-united+kingdom_1651442').read()
html = BeautifulSoup(raw_html, 'html.parser')
for p in html.select('tblrow record'):
print(p)
I am getting an illegal argument on open.
python
asked Nov 12 '18 at 14:15
Ceri WestcottCeri Westcott
32
add a comment |
I am trying to scrape some data from the ancestry, I have a .net background but thought i'd try a bit of python for a project.
I'm falling at the first step, Firstly i am trying to open this page and then just print out the rows.
from requests import get
from requests.exceptions import RequestException
from contextlib import closing
from bs4 import BeautifulSoup
raw_html = open('https://www.ancestry.co.uk/search/collections/britisharmyservice/?
birth=_merthyr+tydfil-wales-united+kingdom_1651442').read()
html = BeautifulSoup(raw_html, 'html.parser')
for p in html.select('tblrow record'):
print(p)
I am getting an illegal argument on open.
python
asked Nov 12 '18 at 14:15
Ceri WestcottCeri Westcott
32
3
openis used for files, not webpages. You probably meant to userequests.get.
– Arnav Borborah
Nov 12 '18 at 14:17
Ah, the tutorial I was following has it wrong then.
– Ceri Westcott
Nov 12 '18 at 14:19
add a comment |
I am trying to scrape some data from the ancestry, I have a .net background but thought i'd try a bit of python for a project.
I'm falling at the first step, Firstly i am trying to open this page and then just print out the rows.
from requests import get
from requests.exceptions import RequestException
from contextlib import closing
from bs4 import BeautifulSoup
raw_html = open('https://www.ancestry.co.uk/search/collections/britisharmyservice/?
birth=_merthyr+tydfil-wales-united+kingdom_1651442').read()
html = BeautifulSoup(raw_html, 'html.parser')
for p in html.select('tblrow record'):
print(p)
I am getting an illegal argument on open.
python
asked Nov 12 '18 at 14:15
Ceri WestcottCeri Westcott
32
I am trying to scrape some data from the ancestry, I have a .net background but thought i'd try a bit of python for a project.
I'm falling at the first step, Firstly i am trying to open this page and then just print out the rows.
from requests import get
from requests.exceptions import RequestException
from contextlib import closing
from bs4 import BeautifulSoup
raw_html = open('https://www.ancestry.co.uk/search/collections/britisharmyservice/?
birth=_merthyr+tydfil-wales-united+kingdom_1651442').read()
html = BeautifulSoup(raw_html, 'html.parser')
for p in html.select('tblrow record'):
print(p)
I am getting an illegal argument on open.
python
python
asked Nov 12 '18 at 14:15
Ceri WestcottCeri Westcott
32
asked Nov 12 '18 at 14:15
Ceri WestcottCeri Westcott
32
asked Nov 12 '18 at 14:15
Ceri WestcottCeri Westcott
32
asked Nov 12 '18 at 14:15
Ceri WestcottCeri Westcott
32
asked Nov 12 '18 at 14:15
Ceri WestcottCeri Westcott
32
32
3
openis used for files, not webpages. You probably meant to userequests.get.
– Arnav Borborah
Nov 12 '18 at 14:17
Ah, the tutorial I was following has it wrong then.
– Ceri Westcott
Nov 12 '18 at 14:19
add a comment |
3
openis used for files, not webpages. You probably meant to userequests.get.
– Arnav Borborah
Nov 12 '18 at 14:17
Ah, the tutorial I was following has it wrong then.
– Ceri Westcott
Nov 12 '18 at 14:19
3
3
open is used for files, not webpages. You probably meant to use requests.get.– Arnav Borborah
Nov 12 '18 at 14:17
open is used for files, not webpages. You probably meant to use requests.get.– Arnav Borborah
Nov 12 '18 at 14:17
Ah, the tutorial I was following has it wrong then.
– Ceri Westcott
Nov 12 '18 at 14:19
Ah, the tutorial I was following has it wrong then.
– Ceri Westcott
Nov 12 '18 at 14:19
add a comment |
1 Answer
1
active
oldest
votes
According to documentation, open is used to:
Open [a] file and return a corresponding file object.
As such, you cannot use it for downloading the HTML contents of a webpage. You probably meant to use requests.get as follows:
raw_html = get('https://www.ancestry.co.uk/search/collections/britisharmyservice/?
birth=_merthyr+tydfil-wales-united+kingdom_1651442').text
# .text gets the raw text of the response
# (http://docs.python-requests.org/en/master/api/#requests.Response.text)
Here are a few recommendation to improve your code as well:
requests.getprovides many useful parameters, one of them beingparams, which allows you to provide the URL parameters in the form of a Python dictionary.- If you need to verify whether the request was successful before accessing its text, then just check if the returned
response.status_code == requests.codes.ok. This only covers status code 200, but if you need more codes, thenresponse.raise_for_statusshould be helpful.
answered Nov 12 '18 at 14:37
Arnav BorborahArnav Borborah
8,04722347
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53264020%2finvalid-argument-in-open-method-for-web-scraping%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
According to documentation, open is used to:
Open [a] file and return a corresponding file object.
As such, you cannot use it for downloading the HTML contents of a webpage. You probably meant to use requests.get as follows:
raw_html = get('https://www.ancestry.co.uk/search/collections/britisharmyservice/?
birth=_merthyr+tydfil-wales-united+kingdom_1651442').text
# .text gets the raw text of the response
# (http://docs.python-requests.org/en/master/api/#requests.Response.text)
Here are a few recommendation to improve your code as well:
requests.getprovides many useful parameters, one of them beingparams, which allows you to provide the URL parameters in the form of a Python dictionary.- If you need to verify whether the request was successful before accessing its text, then just check if the returned
response.status_code == requests.codes.ok. This only covers status code 200, but if you need more codes, thenresponse.raise_for_statusshould be helpful.
answered Nov 12 '18 at 14:37
Arnav BorborahArnav Borborah
8,04722347
add a comment |
According to documentation, open is used to:
Open [a] file and return a corresponding file object.
As such, you cannot use it for downloading the HTML contents of a webpage. You probably meant to use requests.get as follows:
raw_html = get('https://www.ancestry.co.uk/search/collections/britisharmyservice/?
birth=_merthyr+tydfil-wales-united+kingdom_1651442').text
# .text gets the raw text of the response
# (http://docs.python-requests.org/en/master/api/#requests.Response.text)
Here are a few recommendation to improve your code as well:
requests.getprovides many useful parameters, one of them beingparams, which allows you to provide the URL parameters in the form of a Python dictionary.- If you need to verify whether the request was successful before accessing its text, then just check if the returned
response.status_code == requests.codes.ok. This only covers status code 200, but if you need more codes, thenresponse.raise_for_statusshould be helpful.
answered Nov 12 '18 at 14:37
Arnav BorborahArnav Borborah
8,04722347
add a comment |
According to documentation, open is used to:
Open [a] file and return a corresponding file object.
As such, you cannot use it for downloading the HTML contents of a webpage. You probably meant to use requests.get as follows:
raw_html = get('https://www.ancestry.co.uk/search/collections/britisharmyservice/?
birth=_merthyr+tydfil-wales-united+kingdom_1651442').text
# .text gets the raw text of the response
# (http://docs.python-requests.org/en/master/api/#requests.Response.text)
Here are a few recommendation to improve your code as well:
requests.getprovides many useful parameters, one of them beingparams, which allows you to provide the URL parameters in the form of a Python dictionary.- If you need to verify whether the request was successful before accessing its text, then just check if the returned
response.status_code == requests.codes.ok. This only covers status code 200, but if you need more codes, thenresponse.raise_for_statusshould be helpful.
answered Nov 12 '18 at 14:37
Arnav BorborahArnav Borborah
8,04722347
According to documentation, open is used to:
Open [a] file and return a corresponding file object.
As such, you cannot use it for downloading the HTML contents of a webpage. You probably meant to use requests.get as follows:
raw_html = get('https://www.ancestry.co.uk/search/collections/britisharmyservice/?
birth=_merthyr+tydfil-wales-united+kingdom_1651442').text
# .text gets the raw text of the response
# (http://docs.python-requests.org/en/master/api/#requests.Response.text)
Here are a few recommendation to improve your code as well:
requests.getprovides many useful parameters, one of them beingparams, which allows you to provide the URL parameters in the form of a Python dictionary.- If you need to verify whether the request was successful before accessing its text, then just check if the returned
response.status_code == requests.codes.ok. This only covers status code 200, but if you need more codes, thenresponse.raise_for_statusshould be helpful.
answered Nov 12 '18 at 14:37
Arnav BorborahArnav Borborah
8,04722347
answered Nov 12 '18 at 14:37
Arnav BorborahArnav Borborah
8,04722347
answered Nov 12 '18 at 14:37
Arnav BorborahArnav Borborah
8,04722347
answered Nov 12 '18 at 14:37
Arnav BorborahArnav Borborah
8,04722347
8,04722347
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53264020%2finvalid-argument-in-open-method-for-web-scraping%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
3
openis used for files, not webpages. You probably meant to userequests.get.– Arnav Borborah
Nov 12 '18 at 14:17
Ah, the tutorial I was following has it wrong then.
– Ceri Westcott
Nov 12 '18 at 14:19