Blog

Dear SEO Team,

5 of the batch 2’s questionnaires are now open. Please complete these questionnaires by the end of Friday. Over the weekend, I will analyze if there is improvement in the consistency of the ratings after giving you all the indication of the ratings.

As before, please give your best assessment for each search term. Your effort is important for the success of the project.

URL: https://research.viva.ws/
Username: Your first name in small letters
Password: Your assigned password or the password assigned to you previously

Thank you very much!

Dear SEO Team,

In the previous Project Update I mentioned that there is good quality in your responses but there is an issue of consistency of your ratings as a team.  For example, some of you gave too many 1’s and 7’s while some of you hardly gave any of the two.  I have discussed with Vincent and Venus and we decided to give a better indication of what the ratings 1 to 7 mean.  Such indication is summarized in the table below:

If your assessment does not fit into any indication, choose the rating that you consider having similar SEO value as your assessment.  You do not need to print the indication out for reference because it is now repeated in every single question in the questionnaire for your convenience.

In our review, we also found that some of you did not look at the Google Ads metrics.  It is understood that some of you may not use such information in your daily work, but as I learned from your colleagues such data show how popular the search terms are (as measured by impressions) among Google users which could affect its SEO value.  For the sake of adding convenience to those who use such data, the data are now shown on the questionnaire itself.  You may still want to download the Excel file to see the total picture of the campaigns as you may want to compare the metrics of a search term with the rest of the campaign.  You may also see the data of the selected search terms by clicking the “Show Performance” and “Show Conversion” buttons.

To ensure the arrangement would work better, I am going to release the first half of the second batch today.  A separate email will be sent to you when the batch is open.  Please complete the 5 questionnaires available to you by Friday.  Please note than not all 12 of you have the same 5 questionnaires.

Thank you very much!

Today I met Dr Peter Kwan again over breakfast to update him about the progress of the project and to ask for advice from him. As always he gave me invaluable advice on the project. I am grateful for that.

Dear SEO Team,

In the last email I mentioned that the new batch of questionnaires was to release on Monday. After the review of your submitted questionnaires I decided to postpone it to next week.

Vincent and I have looked at the responses you gave to the first batch. We concluded that most of you have provided quality input, but we also found that there is not enough consistency in the ratings by different respondents. It makes sense that we give a better indication of what ratings 1 to 7 mean so that there is better consistency and thus better input for the machine learning models. This will be further discussed with Venus tomorrow, and we will find an opportunity to communicate our conclusions to you as soon as possible.

That’s the update. Stay healthy, and we’ll talk again soon!

At the beginning of the week I analyzed the data collected from Batch 1. I looked at the dataset and I was not exactly certain how I should feel. The standard deviation of the ratings for each question is quite high and the patterns of the ratings from some of the experts confused me. I took those questionnaires which I thought was problematic and discussed with Vincent. Vincent convinced me that most questionnaires are fine, and we concluded that we should give the respondents better indication of each rating score could mean. So I am delaying the release of the second batch of questionnaire to ensure high quality input to the machine learning algorithms. After the discussion with Venus, the leader of the SEO Team, I am feeling much more confident about the project.

Dear SEO Team,

A number of questionnaires were completed before last week's deadline. Some of you are catching up during the Chinese New Year holidays and I appreciate that.

I have done some initial analysis with the submitted questionnaire data during the holidays. It is apparent that we need a lot more data to make the model meaningful.

I have made some changes during the holidays about the SEO data files of every questionnaire:
Although in your daily work the search terms reports from Google are not nicely organized when downloaded, I want to make things easier for this exercise. You will see that the data files are easier to refer to when you download them.
Now only data of the selected campaigns are included.

If you have not finished Batch 1 yet, please do so immediately. The next batch will be released on Monday.

Thank you very much!

Happy new year is to me and the project!

Despite not all members of the SEO team has started, some of them have taken time out of their busy schedule to fill the questionnaires, including during the last day of the last lunar year and the first day of the new lunar year.

I want to thank them for their dedication. I am hoping that the project will not fall behind schedule too much.

Today I visited YouFind and spent pretty much the whole day there retrieving search term report data from Google Ads. Data of 55 clients were retrieved. If everything was done right then this part of data collection is completely complete!!!

Thanks to Vincent Lin for helping me in this area.

Now the challenge on data collection is about search term rating.

Dear SEO Team,

The second half of the first batch has been released since Friday evening. Now we have the full first batch online. That is all we will have before the Chinese New Year.

Last week I asked you to complete the first 4 questionnaires. I understand that many of you are very busy before the Chinese New Year and only some of you could complete all four of them. In order to catch up with the deadline set by the university, please complete the 9 questionnaires as much as possible before the Chinese New Year holidays. I will use data collected to do a pilot study over the Chinese New Year holidays.

Again, I would like to emphasize that your input is key to the success of the project. Your help will be very much appreciated.

To make it more convenient for you, here are the relevant links:

• Questionnaire webapp: https://research.viva.ws/
• Password set/reset link: https://research.viva.ws/password-reset/ (*If you do not see the password reset email after requesting for password reset, please see if you get it in your spam email folder.)

If you have any questions, please call me or WhatsApp me at 92187210. You may also talk to Morus or Nicholas if you prefer.

Thank you very much, and happy a prosperous Chinese New Year!

Wikipedia: Murphy's law is an adage or epigram that is typically stated as: "Anything that can go wrong will go wrong".

Of course, Murphy's law applies here! Testing was done thoroughly for the "normal" process, but numerous non-normal things happened. For example, I did not expect there is a deadline for new accounts to use a predefined link to set its password, or that a reset password link does not work properly under a multi-lingual environment. Could I have prevented those problems? Maybe some of them, but not all. It could be completely random thoughts if I had thought about those cases in advance. But next time I will definitely look into such cases for similar applications.

Yesterday, the login information was sent to YouFind's experts. The questionnaires are officially open. Clarifications about the questionnaires and instructions on how the questionnaires are to be filled are also sent to them. It is understood that this is extra work in additional to their normal workload. I hope the schedule can be followed as much as possible.

Today Nicolas Fu and Morus Cheung of YouFind tested the password reset process and completed testing of the first 2 questionnaires. They have asked many valid questions. I found that the previous communications about the project were not clear enough to articulate the objectives of the questionnaire. I can now describe the project in a much clearer way. Thanks to Nicolas and Morus for challenging me. I'm more confident that this exercise will be fruitful.