In the previous posts, I covered how to scrape some data (like a stock price) from a website. To get a workable dataset, we can write some code to continually loop, and collect that same data at a fixed interval.<\/p>\n
The code below does this. A few points. (1) Python uses indentation as part of the syntax. After starting a loop (the while 1==1: statement below) or a conditional (the if XXX==YYY statement below), everything you want looping or conditionally done has to be indented. (2) the while 1==1 line simply says keep doing this … forever. Since 1 will always be equal to 1. and (3) the if statement below checks if the current minute is divisible by 5 and runs the scraping code if it is. You can change the interval by changing 5 to another number, or using the now.second or now.hour numbers.<\/p>\n
from selenium import webdriver
\nimport datetime
\nimport time
\nfrom multiprocessing import Pool,TimeoutError
\nimport urllib.request
\nimport re
\nfrom urllib.error import URLError, HTTPError<\/p>\nwhile 1==1:
\nnow = datetime.datetime.now()
\nif now.minute\/5 == int(now.minute\/5):
\ndriverspy = webdriver.Chrome()
\ndriverspy.get(‘https:\/\/finance.yahoo.com\/quote\/SPY?p=SPY’)
\nsourcespy = driverspy.page_source
\nnow = datetime.datetime.now()
\nfound = re.search(‘”52″>(\\d+\\.\\d+)<\/span>’, sourcespy).group(1)
\nprint(“Time:”+str(now.hour)+”:”+str(now.minute)+”:”+str(now.second)+” Price:”+str(found))
\ntime.sleep(75)
\ndriverspy.quit()<\/p><\/blockquote>\nWhile the code runs, you’ll get output that looks like the following. You can then either copy paste this to a CSV file or use Python code to export it in order to start building a dataset.<\/p>\n
Time:12:15:20 Price:302.10\r\nTime:12:20:8 Price:302.08\r\nTime:12:25:19 Price:302.05\r\nTime:12:30:20 Price:302.07\r\nTime:12:35:9 Price:302.17\r\nTime:12:40:9 Price:302.09\r\nTime:12:45:28 Price:302.22\r\nTime:12:50:28 Price:302.24\r\nTime:12:55:16 Price:302.26\r\nTime:13:0:8 Price:302.18\r\nTime:13:5:9 Price:302.01\r\nTime:13:10:8 Price:301.96\r\nTime:13:15:28 Price:302.01\r\nTime:13:20:29 Price:302.04\r\nTime:13:25:8 Price:301.96\r\nTime:13:30:20 Price:301.96\r\nTime:13:35:19 Price:302.10\r\nTime:13:40:28 Price:302.27\r\nTime:13:45:20 Price:302.24\r\nTime:13:50:8 Price:302.21\r\nTime:13:55:8 Price:302.19\r\nTime:14:0:8 Price:302.16<\/pre>\n","protected":false},"excerpt":{"rendered":"In the previous posts, I covered how to scrape some data (like a stock price) from a website. To get a workable dataset, we can write some code to continually loop, and collect that same data at a fixed interval. The code below does this. A few points. (1) Python uses indentation as part of […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[9,10],"tags":[],"_links":{"self":[{"href":"http:\/\/www.sugata.in\/index.php\/wp-json\/wp\/v2\/posts\/256"}],"collection":[{"href":"http:\/\/www.sugata.in\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.sugata.in\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.sugata.in\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.sugata.in\/index.php\/wp-json\/wp\/v2\/comments?post=256"}],"version-history":[{"count":3,"href":"http:\/\/www.sugata.in\/index.php\/wp-json\/wp\/v2\/posts\/256\/revisions"}],"predecessor-version":[{"id":261,"href":"http:\/\/www.sugata.in\/index.php\/wp-json\/wp\/v2\/posts\/256\/revisions\/261"}],"wp:attachment":[{"href":"http:\/\/www.sugata.in\/index.php\/wp-json\/wp\/v2\/media?parent=256"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.sugata.in\/index.php\/wp-json\/wp\/v2\/categories?post=256"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.sugata.in\/index.php\/wp-json\/wp\/v2\/tags?post=256"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}